Commit Graph

4854 Commits

Author SHA1 Message Date
Lisa Cawley 859c6441b3 [DOCS] Adds PKI delegation.enabled example (#53030) 2020-03-04 14:59:45 -08:00
Ross Wolf a5e82d7fd6
EQL: Add explicit 'any where ...' handling (#52526) 2020-03-04 10:11:03 -07:00
Nik Everett 609c61f75c
Formalize usage stats for analytics (backport of #52966) (#53077)
This moves the usage statistics gathering from the `AnalyticsPlugin`
into an `AnalyicsUsage`, removing the static state. It also checks the
license level when parsing all analytics aggregations. This is how we
were checking them before but we did it in an easy to forget way. This
way is slightly simpler, I think.
2020-03-04 10:29:11 -05:00
Martijn van Groningen 3fa5395ac8
Use correct issue number: #52453 2020-03-04 16:17:55 +01:00
Martijn van Groningen 2e325e24cb
Mute testMonitorClusterHealth test (#53109)
Relates to #36782
2020-03-04 16:08:19 +01:00
Martijn van Groningen b77f6746d1
unmute watcher single node test case
relates to #36782
2020-03-04 15:25:17 +01:00
Aleksandr Maus b47bffba24
EQL: consistent naming for event type vs event category (#53073) (#53090)
Related to https://github.com/elastic/elasticsearch/issues/52941
2020-03-04 08:02:38 -05:00
Marios Trivyzas e180e2738a
SQL: [Tests] Add tests for optimization of aliased expressions (#53048)
Add a unit test to verify that the optimization of expression
(e.g. COALESCE) is applied to all instances of the expression:
SELECT, WHERE, GROUP BY and HAVING.

Relates to #35270

(cherry picked from commit 2ceedc7f2019fad92cd86679af1a9c6fa594aa8d)
2020-03-04 11:48:06 +01:00
Marios Trivyzas 1d5c842700 SQL: Fix column size for IP data type (#53056)
Set size/displaySize to 45 which is the maximum string for
an IP (v6), since IPs are returned as strings.

Fixes: #52762

(cherry picked from commit 815f01747a4d54a274ca248af6fc08e5ea0728c1)
2020-03-04 10:36:44 +01:00
Mark Vieira 4b528d97ad
Consolidate duplication of BWC testing task setup in script plugin (#53079)
(cherry picked from commit 33fc8e7ebfac8d47a5f9f026b3836bb47bea141a)
2020-03-03 14:43:02 -08:00
Jay Modi c610e0893d
Introduce system index APIs for Kibana (#53035)
This commit introduces a module for Kibana that exposes REST APIs that
will be used by Kibana for access to its system indices. These APIs are wrapped
versions of the existing REST endpoints. A new setting is also introduced since
the Kibana system indices' names are allowed to be changed by a user in case
multiple instances of Kibana use the same instance of Elasticsearch.

Additionally, the ThreadContext has been extended to indicate that the use of
system indices may be allowed in a request. This will be built upon in the future
for the protection of system indices.

Backport of #52385
2020-03-03 14:11:36 -07:00
Andrei Stefan 9ad9ad7a6b
SQL: update SqlNodeSubclassTests list of min-two-parameters functions list (#53045) (#53058)
(cherry picked from commit c741e49d9f5e7b78c1a78e1af97eb19354fe6864)
2020-03-03 19:37:37 +02:00
Adrien Grand cb868d2f5e
Introduce a `constant_keyword` field. (#49713) (#53024)
This field is a specialization of the `keyword` field for the case when all
documents have the same value. It typically performs more efficiently than
keywords at query time by figuring out whether all or none of the documents
match at rewrite time, like `term` queries on `_index`.

The name is up for discussion. I liked including `keyword` in it, so that we
still have room for a `singleton_numeric` in the future. However I'm unsure
whether to call it `singleton`, `constant` or something else, any opinions?

For this field there is a choice between
 1. accepting values in `_source` when they are equal to the value configured
    in mappings, but rejecting mapping updates
 2. rejecting values in `_source` but then allowing updates to the value that
    is configured in the mapping
This commit implements option 1, so that it is possible to reindex from/to an
index that has the field mapped as a keyword with no changes to the source.

Backport of #49713
2020-03-03 16:01:47 +01:00
Yang Wang 70814daa86
Allow _rollup_search with read privilege (#52043) (#53047)
Currently _rollup_search requires manage privilege to access. It should really be
a read only operation. This PR changes the requirement to be read indices privilege.

Resolves: #50245
2020-03-03 22:29:54 +11:00
Martijn van Groningen 510db25dd0
Simplify watcher indexing listener.(#53046)
Backport: #52627

Add watcher to trigger server after index operation has succeeded,
instead of adding a watch to trigger service before
the actual index operation has performed on the shard level.

This logic is simpler to reason about in the case that a failure
does occur during the execution of an index operation on
the shard level.

Relates to #52453, but I think doesn't fix it, but makes it easier
to debug.
2020-03-03 11:01:57 +01:00
Hendrik Muhs 844f350774 [Transform] restructure transform yaml tests (#52956)
restructure transform yaml tests to run cleanup in teardown phase

relates #52428
2020-03-03 10:31:22 +01:00
Hendrik Muhs d9258e210e [Transform] fix sporadic race condition in TransformUsageIT (#52946)
relax the test for trigger count

fixes #52931
2020-03-03 10:27:36 +01:00
Costin Leau 712e0c05cd EQL: Add implicit ordering on timestamp (#53004)
QL: Move Sort base class from SQL to QL
(cherry picked from commit 798015b7bbd565e9c4222724614baeb432c7c2b3)
2020-03-02 22:41:36 +02:00
Mark Vieira f8396e8d15
Mute RunDataFrameAnalyticsIT.testStopOutlierDetectionWithEnoughDocumentsToScroll
Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-03-02 09:21:55 -08:00
Mark Vieira 5b5e92c71d
Mute NodeSubclassTests.testReplaceChildren
Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-03-02 09:21:54 -08:00
Lisa Cawley 5a293b29a1 [DOCS] Add missing anchors in cron expressions (#52960) 2020-03-02 09:07:22 -08:00
Lisa Cawley 4fbe1b0550
[DOCS] Adds cat anomaly detectors API (#52866) (#52970) 2020-03-02 07:28:55 -08:00
Hendrik Muhs a328a8eaf1
[7.x][Transform] implement node.transform to control where to… (#52998)
implement transform node attributes to disable transform on certain nodes and
test which nodes are allowed to do remote connections

closes #52200
closes #50033
closes #48734

backport #52712
2020-03-02 16:10:57 +01:00
Aleksandr Maus 89ed857c79
EQL: Change request parameter query to filter and rule to query (#52971) (#53006)
Related to https://github.com/elastic/elasticsearch/issues/52911
2020-03-02 09:26:23 -05:00
Andrei Stefan 6fecc1db84
Issue a different error message in case an index doesn't have a mapping (#52967) (#53003)
(cherry picked from commit a0bd83a0579cf196a1d727de2a46b3b101d5a73b)
2020-03-02 14:04:49 +02:00
Andrei Stefan 69383acecf
Define list of Nodes that have minimum two children in tests (#52957) (#52994)
(cherry picked from commit c1e43e694f02edf3e197abbab7c21008c022b516)
2020-03-02 11:26:50 +02:00
Hendrik Muhs 49f41d127b [Transform] fix NPE in derive stats if shouldStopAtNextCheckpo… (#52940)
fixes a NPE in _stats in case shouldStopAtNextCheckpoint is set.
2020-03-02 08:11:01 +01:00
Martijn van Groningen d102158e6f
Improve closing mock webserver when failed to start (#52943)
Fix NPE when closing a webserver that hasn't started correctly.

This can happen when ssl context isn't initialized. The server instance is then never set,
which causes an NPE that masks the actual failure.

Example stacktrace that would mask an actual failure:

```
java.lang.NullPointerException
	at org.elasticsearch.test.http.MockWebServer.close(MockWebServer.java:271)
	at org.elasticsearch.xpack.watcher.test.integration.HttpSecretsIntegrationTests.cleanup(HttpSecretsIntegrationTests.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
```
2020-03-02 07:19:08 +01:00
Nhat Nguyen e6755afeeb
Upgrade to Lucene 8.5.0-snapshot-c4475920b08 (#52950) (#52977)
To give LUCENE-9228 more CI cycles
2020-02-29 09:29:16 -05:00
Dimitris Athanasiou 85b4e45093
[7.x]ML] Parse and report memory usage for DF Analytics (#52778) (#52980)
Adds reporting of memory usage for data frame analytics jobs.
This commit introduces a new index pattern `.ml-stats-*` whose
first concrete index will be `.ml-stats-000001`. This index serves
to store instrumentation information for those jobs.

Backport of #52778 and #52958
2020-02-29 13:03:40 +02:00
Luca Cavanna 090bdf69c0
Mute NodeSubclassTests#testReplaceChildren (#52952)
Relates #52951
2020-02-28 16:13:17 +01:00
Andrei Stefan c3a167830f
SQL: refactor In predicate moving it to QL project (#52870) (#52938)
* Move In, InPipe and InProcessor out of SQL to the common QL project.
* Move tests classes to the QL project.
* Create SQL dedicated In class to handle SQL specific data types.
* Update SQL classes to use the InPipe and InProcessor QL classes.
* Extract common Foldables methods in QL project.
* Be more explicit when folding and converting a foldable value, by
removing most of the code inside Foldables class.

(cherry picked from commit 7425042f86f66df8c207c5e96f9b9848bda2b4c3)
2020-02-28 14:04:10 +02:00
Costin Leau a674085903 EQL: Disable field extraction for returned events (#52884)
Return the whole source of matching events

(cherry picked from commit 79ca586ab1d89d645fb58142b82202f14ce5d361)
2020-02-28 13:48:15 +02:00
Yang Wang 82553524af
Respect runas realm for ApiKey security operations (#52178) (#52932)
When user A runs as user B and performs any API key related operations,
user B's realm should always be used to associate with the API key.
Currently user A's realm is used when getting or invalidating API keys
and owner=true. The PR is to fix this bug.

resolves: #51975
2020-02-28 10:53:52 +11:00
Nik Everett 866b08716c
Fix test for top_metrics (#52927)
I added the wrong skips and the wrong error message. Ooops.
2020-02-27 18:30:37 -05:00
Nik Everett 1d1956ee93
Add size support to `top_metrics` (backport of #52662) (#52914)
This adds support for returning the top "n" metrics instead of just the
very top.

Relates to #51813
2020-02-27 16:12:52 -05:00
Benjamin Trent 19a6c5d980
[7.x] [ML][Inference] Add support for multi-value leaves to the tree model (#52531) (#52901)
* [ML][Inference] Add support for multi-value leaves to the tree model (#52531)

This adds support for multi-value leaves. This is a prerequisite for multi-class boosted tree classification.
2020-02-27 14:05:28 -05:00
Benjamin Trent eac38e9847
[ML] Add indices_options to datafeed config and update (#52793) (#52905)
This adds a new configurable field called `indices_options`. This allows users to create or update the indices_options used when a datafeed reads from an index.

This is necessary for the following use cases:
 - Reading from frozen indices
 - Allowing certain indices in multiple index patterns to not exist yet

These index options are available on datafeed creation and update. Users may specify them as URL parameters or within the configuration object.

closes https://github.com/elastic/elasticsearch/issues/48056
2020-02-27 13:43:25 -05:00
Henning Andersen 09fe4b42db Disable ILM history in x-pack rest tests (#52868)
The ILM history index can be delayed created from one test into the
next, which can cause issues for tests using `_all`.

Closes #52209
2020-02-27 17:20:33 +01:00
David Kyle d8bdf31110 Revert "Mute RunDataFrameAnalyticsIT.testOutlierDetectionStopAndRestart"
This reverts commit ad3a3b1af9.
2020-02-27 12:38:13 +00:00
David Kyle 6e5e64559a
Unwrap cause from remote ActionTransportExceptions (#52842) (#52878)
And log the cause
2020-02-27 11:58:28 +00:00
István Zoltán Szabó 4a33352a94 [DOCS] Adds cat trained model API documentation (#52824) 2020-02-27 12:54:11 +01:00
Costin Leau 40bc06f6ad EQL: Hook engine to Elasticsearch (#52828)
Add query execution and return actual results returned from
Elasticsearch inside the tests

(cherry picked from commit 3e039282bf991af87604a6d4f8eada19d5e33842)
2020-02-27 11:22:22 +02:00
Yang Wang 14c21aedd2
Simplify ml license checking with XpackLicenseState internals (#52684) (#52863)
This change removes TrainedModelConfig#isAvailableWithLicense method with calls to
XPackLicenseState#isAllowedByLicense.

Please note there are subtle changes to the code logic. But they are the right changes:
* Instead of Platinum license, Enterprise license nows guarantees availability.
* No explicit check when the license requirement is basic. Since basic license is always available, this check is unnecessary.
* Trial license is always allowed.
2020-02-27 14:14:16 +11:00
Yang Wang f5c4e92558
Refactor license checking (#52118) (#52859)
Improve code resuse and readility. Add convenience checking method which
covers most use cases without having to pass many boolean arguments.
2020-02-27 13:04:19 +11:00
Jake Landis b4179a8814
[7.x] Refactor watcher tests (#52799) (#52844)
This PR moves the majority of the Watcher REST tests under
the Watcher x-pack plugin.

Specifically, moves the Watcher tests from:
x-pack/plugin/test
x-pack/qa/smoke-test-watcher
x-pack/qa/smoke-test-watcher-with-security
x-pack/qa/smoke-test-monitoring-with-watcher

to:
x-pack/plugin/watcher/qa/rest (/test and /qa/smoke-test-watcher)
x-pack/plugin/watcher/qa/with-security
x-pack/plugin/watcher/qa/with-monitoring

Additionally, this disables Watcher from the main
x-pack test cluster and consolidates the stop/start logic
for the tests listed.

No changes to the tests (beyond moving them) are included.

3rd party tests and doc tests (which also touch Watcher)
are not included in the changes here.
2020-02-26 15:57:10 -06:00
Jay Modi 07ef8ccff4
Allow dynamic updates for index.hidden setting (#52837)
This commit changes the `index.hidden` setting from being final to a
dynamic setting. While the setting being final allows for easier
reasoning about an index, making this setting update-able has more
benefits in that we can upgrade existing indices to be hidden and it
will enable future features that would dynamically make indices hidden.

Backport of #52772
2020-02-26 11:46:29 -07:00
Nik Everett bfaa487757
Switch pipeline agg parsing to ContextParser (#52776) (#52832)
We've pretty well settled on `ContextParser` for a generic interface to
`ObjectParser`-like-things. This switches the interface used for
building parsing pipeline aggregations to `ContextParser` which saves a
couple of little wrappers around `ObjectParser`.
2020-02-26 12:57:20 -05:00
Lisa Cawley b788ec7157 [DOCS] Adds cat datafeeds API (#52738) 2020-02-26 09:28:57 -08:00
Ioannis Kakavas 2d01c005ba
Update commons-collections test dependency to 3.2.2 (#52808) (#52817)
This is only a test dependency but it trips scanners so upgrade to
3.2.2 which doesn't suffer from the issues mentioned in i.e.
https://snyk.io/vuln/SNYK-JAVA-COMMONSCOLLECTIONS-472711
2020-02-26 17:03:45 +02:00