1460 Commits

Author SHA1 Message Date
Tim Sullivan
723d894753 [Monitoring/Cluster Alerts] Fix the email message when cluster license expiration is resolved (elastic/x-pack-elasticsearch#2557)
* [Monitoring/Cluster Alerts] Fix the email message for cluster license expiration resolved

* fix making payload.message show only when new

Original commit: elastic/x-pack-elasticsearch@6d54b02913
2017-09-20 15:44:14 -07:00
Igor Motov
59892ba654 SQL: Fix KnownActionsTests after introduction of the _sql/translate endpoint
Original commit: elastic/x-pack-elasticsearch@a6f2c05a5e
2017-09-20 17:45:58 -04:00
Costin Leau
6cc3c067b7 Introduce _sql/translate endpoint
Builds on RestSqlAction and in fact, extends SqlAction to keep up
with future request settings.

Original commit: elastic/x-pack-elasticsearch@7bbef4bdff
2017-09-20 21:48:53 +03:00
Igor Motov
4964239f4c Tests: Remove AwaitsFix from TemplateUpgraderTests
I believe this problem should have been already fixed by elastic/elasticsearch#26698.

Original commit: elastic/x-pack-elasticsearch@0ec93f2803
2017-09-20 14:12:24 -04:00
Dimitrios Athanasiou
a5e8589b38 [ML][TEST] Add integ test for elastic/x-pack-elasticsearch#2519
Relates elastic/x-pack-elasticsearch#2519

Original commit: elastic/x-pack-elasticsearch@91b38a62bb
2017-09-20 18:59:34 +01:00
Nik Everett
2df8b0c144 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@dff8c22d99
2017-09-20 12:06:27 -04:00
Dimitris Athanasiou
fad98d784f [ML] Align aggregated data extraction to histogram interval (elastic/x-pack-elasticsearch#2553)
When the datafeed uses aggregations and in order to accommodate
derivatives, an extra bucket is queried at the beginning of each
search. In order to avoid visiting the same bucket twice, we need
to search buckets aligned to the histogram interval. This allows
us to steer away from partial buckets, and thus avoid the problem
of dropping or duplicating data.

relates elastic/x-pack-elasticsearch#2519

Original commit: elastic/x-pack-elasticsearch@e03dde5fea
2017-09-20 16:45:06 +01:00
Nik Everett
6ea902f913 SQL: NOCOMMIT cleanup
Removes a few NOCOMMITs that are tracked other places and updates
a few with plans on how to work on them.

Original commit: elastic/x-pack-elasticsearch@8d1cfdf4ee
2017-09-20 11:19:05 -04:00
Hendrik Muhs
3c517902f2 re-enable categorization IT (elastic/x-pack-elasticsearch#2534)
re-enables categorization IT after fixing https://github.com/elastic/machine-learning-cpp/issues/279 upstream

Original commit: elastic/x-pack-elasticsearch@398a668125
2017-09-20 11:39:37 +02:00
Yannick Welsch
8648153f0e Deguice ActionFilter (elastic/x-pack-elasticsearch#2533)
Companion PR to elastic/elasticsearch#26691

Original commit: elastic/x-pack-elasticsearch@3fceb54809
2017-09-20 10:30:23 +02:00
Tal Levy
8b1021ccad blacklist two license rest tests when build.snapshot=false (elastic/x-pack-elasticsearch#2559)
There are two rest tests that rely on hardcoded license
signatures that use the dev public key. These tests fail
when tests are run with build.snapshot=false. This Commit
blacklists these two tests in that scenario

relates elastic/x-pack-elasticsearch#2527.

Original commit: elastic/x-pack-elasticsearch@7581e8d699
2017-09-19 16:56:11 -07:00
Nik Everett
ea66433899 Mark periodically failing test as AwaitsFix
This should reduce our build failure emails. The failure is already
tracked at https://github.com/elastic/x-pack-elasticsearch/issues/2421.
It isn't actively being worked but will be soon.

Original commit: elastic/x-pack-elasticsearch@6baf55de42
2017-09-19 09:59:26 -04:00
David Roberts
67055877a5 [ML] Avoid creating spurious 0 "actual" values for model plot documents (elastic/x-pack-elasticsearch#2535)
Some model plot documents should not have an "actual" value, for example
when no input events were seen for a meean/min/max detector in a particular
bucket.  Prior to this change we would set the "actual" value to 0 for such
model plot documents.  Following this change no "actual" value will be
present in these documents.

Only newly created model plot documents are affected.  Model plot documents
that were incorrectly written in the past will remain wrong forever.

relates elastic/x-pack-elasticsearch#2528

Original commit: elastic/x-pack-elasticsearch@47a7365f59
2017-09-19 10:30:43 +01:00
Tim Vernum
aec2308228 Allow AD realm to perform 'run-as' lookups (elastic/x-pack-elasticsearch#2531)
- Marks the AD Session factory as supporting "lookup" (Refer: elastic/x-pack-elasticsearch@40b07b3)
- Adds "pool.enabled" as a registered setting on AD realm (Refer: elastic/x-pack-elasticsearch@40b07b3)
- Fixes LDAP user lookup that has been broken since 6.x (Refer: elastic/x-pack-elasticsearch@f796949)

Original commit: elastic/x-pack-elasticsearch@62ff6129a1
2017-09-19 14:50:26 +10:00
Nik Everett
641db10605 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@81ed649a9b
2017-09-18 16:33:15 -04:00
Tal Levy
5a090c14c1 convert more request objects to writeable (elastic/x-pack-elasticsearch#2457)
* convert more to writeable

* migrate streamable tests to writeable tests

Original commit: elastic/x-pack-elasticsearch@56794e5760
2017-09-18 13:20:02 -07:00
Suyog Rao
3a9aad5ece [Logstash] Remove version field from config mgmt
relates elastic/x-pack-elasticsearch#2405

Original commit: elastic/x-pack-elasticsearch@5bfd1b7a6d
2017-09-18 12:35:21 -07:00
Simon Willnauer
47214426e9 Use InputStreamStreamInputs validation to limit size of tokens (elastic/x-pack-elasticsearch#2537)
Relates to elastic/elasticsearch#26692
relates elastic/x-pack-elasticsearch#2493

Original commit: elastic/x-pack-elasticsearch@8e23868743
2017-09-18 19:25:08 +02:00
Nik Everett
52ee02da27 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@c25c179ce6
2017-09-18 12:32:46 -04:00
Dimitris Athanasiou
9b25d0edf7 [ML] Correctly ignore extra aggregation buckets (elastic/x-pack-elasticsearch#2530)
The problem here is that the code was ignoring buckets
whose start time was before the start time of the extractor.
However, this is not a good enough condition. For example,
when there are no data in the bucket extra bucket that is
being queried, the first bucket will be the one containing
the start time.

This commit fixes the issue by changing the condition to
ignore buckets before the first bucket that includes the
start time of the extraction.

relates elastic/x-pack-elasticsearch#2519 

Original commit: elastic/x-pack-elasticsearch@15c7d2655f
2017-09-18 12:10:19 +01:00
Tim Vernum
dea82a07a2 Infer KeyStore type from pathname (elastic/x-pack-elasticsearch#2514)
If the keystore type is not explicitly specified, infer it from the filename.
Treats .p12, .pfx and .pkcs12 as being PKCS12, all others as jks.

This will allow certgen to produce PKCSelastic/x-pack-elasticsearch#12 files by default and make it easy to use them as x-pack keystores

Original commit: elastic/x-pack-elasticsearch@fc361f0d87
2017-09-18 14:21:19 +10:00
Chris Earle
24c2c62ca2 [Monitoring] Ignore .marvel* indices (elastic/x-pack-elasticsearch#2520)
Beginning with 7.0, the cleaner service will no longer automatically cleanup .marvel indices regardless of their age.

Original commit: elastic/x-pack-elasticsearch@5b90e6f62a
2017-09-15 13:35:45 -07:00
Tim Brooks
b3914afd30 Reenable TribeWithSecurityIT tests (elastic/x-pack-elasticsearch#2511)
This is related to elastic/x-pack-elasticsearch#1996. These tests were disabled during the bootstrap
password work. They can now be reenabled. Additionally, I made the test
password used in tests consistent.

Original commit: elastic/x-pack-elasticsearch@5b490c8231
2017-09-15 12:50:54 -06:00
Simon Willnauer
96e01dce47 Only require TLS for standard/gold/platinum licenses elastic/x-pack-elasticsearch#2517
relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5213cf24f1
2017-09-15 20:21:15 +02:00
Chris Earle
9c9da2e1e4 [Monitoring] Remove Legacy Monitoring Indices (elastic/x-pack-elasticsearch#2513)
This changes Monitoring's Cleaner Service to remove any legacy Monitoring index that is appropriately old.

This includes any `.marvel-*` index and also the "data" indices used by both Marvel and 5.0 - 5.4 versions of X-Pack monitoring, as well as the legacy alerts index.

Original commit: elastic/x-pack-elasticsearch@8d99f5518b
2017-09-15 08:46:10 -07:00
jaymode
84dd719ab9 remove outdated comment
Original commit: elastic/x-pack-elasticsearch@06a51abb65
2017-09-15 09:03:36 -06:00
jaymode
344603e40f update text in TLSLicenseBootstrapCheck
Original commit: elastic/x-pack-elasticsearch@4ee6827566
2017-09-15 08:56:34 -06:00
Simon Willnauer
4d20586b24 [TEST] add integration test that ensures we reject license upgrades if TLS is not enabled
Original commit: elastic/x-pack-elasticsearch@dfbadb5e5f
2017-09-15 14:47:28 +02:00
Simon Willnauer
c3066d1a51 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@9ce33bc7c3
2017-09-15 09:51:16 +02:00
Simon Willnauer
023bdb72b2 Add common-analysis plugin to several xpack integ tests (elastic/x-pack-elasticsearch#2501)
Several tests miss the common-analysis plugin in the old-style integ
tests causing odd exceptions in the test logs. This adds the missing plugin reference.

relates elastic/x-pack-elasticsearch#2363

Original commit: elastic/x-pack-elasticsearch@e4e6735408
2017-09-15 09:25:36 +02:00
jaymode
8997792875 Test: use TLS for plugin integ tests
Original commit: elastic/x-pack-elasticsearch@99971d7256
2017-09-14 15:57:28 -06:00
Michael Basnight
fa0b854fb6 Update rest-api-spec to use bad_request (elastic/x-pack-elasticsearch#2507)
ref #elastic/elasticsearch#26539

Original commit: elastic/x-pack-elasticsearch@8b79a0769a
2017-09-14 15:59:29 -05:00
Andy Bristol
279c7e14fd [TEST] fix security template version check in rest tests (elastic/x-pack-elasticsearch#2506)
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.

Original commit: elastic/x-pack-elasticsearch@d66145de54
2017-09-14 12:16:20 -07:00
Jay Modi
57de66476c Disable TLS by default (elastic/x-pack-elasticsearch#2481)
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@abc66ec67d
2017-09-14 12:18:54 -06:00
Simon Willnauer
1e14e14571 Prevent licenses to be upgraded to production unless TLS is configured (elastic/x-pack-elasticsearch#2502)
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
2017-09-14 20:14:27 +02:00
Nik Everett
bac9afee7e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@270ab71b19
2017-09-14 11:43:48 -04:00
Nik Everett
5d3f5cc4f8 Support scrolling in SQL's CLI (elastic/x-pack-elasticsearch#2494)
* Move CLI to TransportSqlAction
    * Moves REST endpoint from `/_cli` to `/_sql/cli`
    * Removes the special purpose CLI transport action instead
    implements the CLI entirely on the REST layer, delegating
    all SQL stuff to the same action that backs the `/_sql` REST
    API.
    * Reworks "embedded testing mode" to use a `FilterClient` to
    bounce capture the sql transport action and execute in embedded.
    * Switches CLI formatting from consuming the entire response
    to consuming just the first page of the response and returning
    a `cursor` that can be used to read the next page. That read is
    not yet implemented.
    * Switch CLI formatting from the consuming the `RowSetCursor` to
    consuming the `SqlResponse` object.
    * Adds tests for CLI formatting.
* Support next page in the cli
    * Rename cli's CommandRequest/CommandResponse to
    QueryInitRequest/QueryInitResponse to line up with jdbc
    * Implement QueryPageRequest/QueryPageResponse in cli
    * Use `byte[]` to represent the cursor in the cli. Those bytes
    mean something, but only to the server. The only reasonint that
    the client does about them is "if length == 0 then there isn't a
    next page."
    * Pull common code from jdbc's QueryInitRequest, QueryPageRequest,
    QueryInitResponse, and QueryPageResponse into the shared-proto
    project
        * By implication this switches jdbc's QueryPageRequest to using
     the same cursor implementation as the cli

Original commit: elastic/x-pack-elasticsearch@193586f1ee
2017-09-14 10:26:42 -04:00
Jay Modi
8d8baffe24 Add specific client and user for security index access (elastic/x-pack-elasticsearch#2492)
This change removes security index access from the xpack user by creating its own specific role
and adds a xpack security user that maintains the superuser role so that it can perform all
operations necessary for security.

Original commit: elastic/x-pack-elasticsearch@ad906bc913
2017-09-14 08:09:14 -06:00
David Roberts
104a3a323f [BUILD] Make AWS error message more informative
Original commit: elastic/x-pack-elasticsearch@42cca7ed82
2017-09-14 14:46:48 +01:00
Simon Willnauer
91b57ee63f Add bootstrap check that enforces TLS if a production license is in the local clusterstate (elastic/x-pack-elasticsearch#2499)
This change will enforce transport SSL to be enforced if security is enabled and the
license in the clusterstate is a production license. The cluster state is loaded from
local storage such that we don't need to join a cluster to make these checks. Yet, the cluster
might have already got a different license if the node got disconnected while the license got
downgraded and then TLS got disabled. This corner case requires manual intervention which
we consider ok given the simplicity of this change.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5765b7cd21
2017-09-14 13:52:53 +02:00
Hendrik Muhs
7d19264363 [ML-FC] Branch landing feature/ml (elastic/x-pack-elasticsearch#2500)
integrate forecasting feature branch into master

    - add endpoint xpack/ml/job/forecast to request forecasting on data of ml-jobs
       - current parameters: end time
    - persists forecast results into shared or own index
       - different runs are separated by a 'forecast id'

relates elastic/x-pack-elasticsearch#1838

Original commit: elastic/x-pack-elasticsearch@f9d701a6bc
2017-09-14 12:31:20 +02:00
Simon Willnauer
3b00251a96 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@4a36f0c2be
2017-09-14 07:43:19 +02:00
Jason Tedor
4f3e740ba8 Refactor bootstrap check results and error messages
This commit refactors the X-Pack bootstrap checks to respond to a change
in core Elasticsearch where the checks now return a single result
object.

Relates elastic/x-pack-elasticsearch#2495

Original commit: elastic/x-pack-elasticsearch@230b050529
2017-09-13 21:30:51 -04:00
Nik Everett
858f0b2dac Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@9945382d90
2017-09-13 16:45:27 -04:00
Nik Everett
f15666f82e Fix links in deprecation checks (elastic/x-pack-elasticsearch#2490)
Some links must have moved since we wrote the tests and released
5.6.0.

relates elastic/x-pack-elasticsearch#2488

Original commit: elastic/x-pack-elasticsearch@ebceee7f3d
2017-09-13 16:32:40 -04:00
Simon Willnauer
01a921a8e3 Accept BootstrapContext in xpack (elastic/x-pack-elasticsearch#2486)
This is the xpack side of elastic/elasticsearch#26628

Original commit: elastic/x-pack-elasticsearch@f6c0599ee2
2017-09-13 22:14:29 +02:00
Jay Modi
f30e5c3fee Register the legacy truststore password setting for the PKI realm (elastic/x-pack-elasticsearch#2487)
After the addition of the secure settings in 5.6, the truststore.password setting for the PKI realm
was no longer registered. This would cause new nodes to fail for customers that were upgrading and
had configured a PKI realm with a truststore. This change registers the setting and adds a test to
ensure a realm configuration with the old setting passes validation.

Relates elastic/support-dev-help#2505

Original commit: elastic/x-pack-elasticsearch@54da044a27
2017-09-13 13:11:54 -06:00
Simon Willnauer
0680e41f36 Prevent nodes from joining a non-TLS enabled cluster with a production license (elastic/x-pack-elasticsearch#2484)
This change prevents a node from joining a cluster with a production license (gold, platinum, standard) iff the cluster doesn't have TLS setup. This is mainly a BWC oriented change that prevents joining old 5.x clusters without a TLS setup.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@21f5a58472
2017-09-13 20:40:35 +02:00
Dimitris Athanasiou
99ffbb1cd6 [ML] Add random offset to the maintenance task execution time (elastic/x-pack-elasticsearch#2483)
Currently the maintenance task is executed at 30 minutes past
midnight of each day. In the scenario where multiple clusters
are running on the same hardware infrastructure they all will
be running at the same time, competing for resources.

This commit changes this by adding a random offset to the
execution time which ranges from 0 to 119 minutes. The
minute granularity means that different offsets give at
least 1 minute for the maintenance task to end. Moreover,
the 2 hour window gives enough slots for different offsets
to occur and remains within what most people would think
as "middle of the night".

relates elastic/x-pack-elasticsearch#2273

Original commit: elastic/x-pack-elasticsearch@b538923aca
2017-09-13 14:53:44 +01:00
Dimitris Athanasiou
e4753656bc [ML] Randomize default datafeed query delay (elastic/x-pack-elasticsearch#2475)
Changes the default query delay from 1m to a random
value between 1m and 2m. The motivation is to avoid
having multiple jobs firing their searches at the same
time which may potentially lead to increased load
on the machine.

relates elastic/x-pack-elasticsearch#2472

Original commit: elastic/x-pack-elasticsearch@3224e836fa
2017-09-13 09:12:39 +01:00