Commit Graph

5779 Commits

Author SHA1 Message Date
lcawley f90cfa9afb [DOCS] Add link to ML aggregations page
Original commit: elastic/x-pack-elasticsearch@9137522f9a
2017-06-02 12:25:38 -07:00
Nik Everett a27ded98c6 Test: crank up logging on LicensingTests
We've seen some failures of the LicensingTests that we don't have enough
information to debug:
https://internal-ci.elastic.co/job/elastic+x-pack-elasticsearch+master+periodic/2220/consoleFull

So this cranks up the logging of the bits we expect to see the failures
in.

Original commit: elastic/x-pack-elasticsearch@d28c1051a4
2017-06-02 11:38:26 -04:00
David Roberts 67ddbf1fac [ML] Don't serialise the detector_index field to cluster state (elastic/x-pack-elasticsearch#1628)
Because:

1. It's pointless, as new detector_index values are assigned when an
   analysis_config is parsed
2. It creates a backwards compatibility issue when upgrading from v5.4

Original commit: elastic/x-pack-elasticsearch@2f61aa457e
2017-06-02 16:38:18 +01:00
Chris Earle 8e76265c26 Always Accumulate Transport Exceptions (elastic/x-pack-elasticsearch#1619)
This is the x-pack side of the removal of `accumulateExceptions()` for both `TransportNodesAction` and `TransportTasksAction`.

There are occasional, random failures that occur during API calls that are silently ignored from the caller's perspective, which also leads to weird API responses that have no response and also no errors, which is obviously untrue.

Original commit: elastic/x-pack-elasticsearch@9b57321549
2017-06-02 10:01:21 -04:00
David Roberts babe27afe0 [ML] Add a detector_index field to detectors, and use it for updates (elastic/x-pack-elasticsearch#1613)
Detectors now have a field called detector_index.  This is also now the
field that needs to be supplied when updating a detector.  (Previously
it was simply index, which was confusing.)

When detectors are added to an analysis_config it will reassign
ascending detector_index values starting from 0.  The intention is
never to allow deletion of detectors from an analysis_config, but
possibly to allow disabling them in the future.  This ensures that
detector_index values in results will always tie up with detector_ids
in the detectors that created them.

relates elastic/x-pack-elasticsearch#1275

Original commit: elastic/x-pack-elasticsearch@20a660b07b
2017-06-02 10:26:01 +01:00
Colin Goodheart-Smithe 36b8fe9b29 Adds nodes usage action to known actions list (elastic/x-pack-elasticsearch#1111)
* Adds nodes usage action to known actions list

* Adds name to all rest actions

Original commit: elastic/x-pack-elasticsearch@77ae827a66
2017-06-02 08:46:48 +01:00
Tanguy Leroux 261bf8d78d [Test] LocalExporterTests should wait for exporters to terminate in a finally block (elastic/x-pack-elasticsearch#1581)
At the end of the test, LocalExporterTests checks if no more monitoring
 data are exporter by checking multiple times the last time nodes_stats
 documents were exported, stopping after 10 seconds. It does this in a
 @After annotated method but it would be better to do this in a finally
 block. Also, it should search for node_stats documents only if the
 monitoring indices exist and are searchable to avoid some "all shards
 failed" failures.

Original commit: elastic/x-pack-elasticsearch@90ffb4affd
2017-06-02 09:12:49 +02:00
Lisa Cawley 922a337884 [DOCS] Change POST to PUT in ML API QuickRef (elastic/x-pack-elasticsearch#1617)
Original commit: elastic/x-pack-elasticsearch@036fd4d548
2017-06-01 14:58:28 -07:00
Lisa Cawley 789cd66202 [DOCS] Add details about ML count functions (elastic/x-pack-elasticsearch#1335)
* [DOCS] Add details about ML count functions

* [DOCS] Address feedback in ML count functions

* [DOCS] Clarify ML population analysis in non-zero count functions

Original commit: elastic/x-pack-elasticsearch@24dbeba891
2017-06-01 14:16:14 -07:00
jaymode ff6fa6790e Test: fix kibana user role works in mixed cluster on older versions
The kibana user role in mixed cluster test expects a old cluster test to have run first that would
disable the kibana user, but that test is not executed on 5.1.1 and prior versions. This change
makes a update request an upsert to account for this.

relates elastic/x-pack-elasticsearch#1527

Original commit: elastic/x-pack-elasticsearch@df2945610e
2017-06-01 14:26:57 -06:00
David Kyle e16787ae7b Fix ML upgrade tests
Original commit: elastic/x-pack-elasticsearch@50e114e7eb
2017-06-01 20:26:57 +01:00
David Roberts bf068e9ec3 [ML] Avoid stack traces in the log when deleting jobs (elastic/x-pack-elasticsearch#1615)
Now we've set the option for one type per index it causes a stack trace
in to be logged if we issue a request to delete two documents with
different types.  We only do this to cover the case of documents left
over from v5.4.  We can avoid it by deleting by query using just the
document IDs.

Original commit: elastic/x-pack-elasticsearch@2abffc7d95
2017-06-01 17:10:41 +01:00
jaymode 68ad44c5c2 Test: enable default password migration test for versions > 5.1.1
Versions below 5.1.1 had a bug where the rest set enabled action would trip an assertion and cause
this test to fail so we skip those versions.

relates elastic/x-pack-elasticsearch#1523

Original commit: elastic/x-pack-elasticsearch@fba8ddd7e7
2017-06-01 08:37:39 -06:00
Alexander Reelsen 730cfd7c7a Watcher: Remote WatcherClientProxy/ClientProxy class (elastic/x-pack-elasticsearch#1561)
This commit removes ClientProxy and WatcherClientProxy classes. They
were added in times, where there were issues with guice and circular
dependencies. However there is no guice anymore and on top of that
the classes do not add any value.

We can switch to use a regular client, but have to make sure that
the InternalClient is injected in all the transport actions as those
is able to query data, when security is enabled.

Original commit: elastic/x-pack-elasticsearch@763a79b2f7
2017-06-01 16:30:21 +02:00
David Kyle 34f526b60b [ML] Yml rolling upgrade tests (elastic/x-pack-elasticsearch#1555)
* ML bwc jobs tests

* Test for starting old data feeds

Original commit: elastic/x-pack-elasticsearch@131c7b15eb
2017-06-01 14:29:10 +01:00
Dimitris Athanasiou 15e53280dc [ML] Allow datafeed to start when job is opening (elastic/x-pack-elasticsearch#1611)
The goal of this change is to allow datafeeds to start
when the job is in the opening state. This makes the API
more async and it allows clients like the ML UI to open a
job and start its datafeed without having to manage the
complexity of dealing with timeouts due to the job taking
time to open due to restoring a large state.

In order to achieve this, this commit does a number of things:

  - accepts a start datafeed request when the job is opening
  - adds logic to the DatafeedManager to wait before running the
  datafeed task until the job is opened
  - refactord the datafeed node selection logic into its own class
  - splitd selection issues in critical and non-critical with regard
  to creating the datafeed task
  - refactord the unit tests to make simpler to write & understand
  - adds unit tests for added and modified functionality
  - changes the response when the datafeed cannot be started to
  be a conflict exception

relates elastic/x-pack-elasticsearch#1535

Original commit: elastic/x-pack-elasticsearch@c83196155d
2017-06-01 12:00:46 +01:00
Tim Vernum fe33d8eba4 [Security] Include doc-type in _id for tokens (elastic/x-pack-elasticsearch#1473)
In preparation for the removal of types, new security types like invalidated-tokens are stored in the .security
index under the generic "doc" type, with a query filter on `doc_type`.

In order to avoid id clashes, we also need to use that doc_type as part of the document id.

relates elastic/x-pack-elasticsearch#1300

Original commit: elastic/x-pack-elasticsearch@469724a228
2017-06-01 10:48:52 +10:00
David Roberts 6484f812c0 [ML] Change the root_cause of error responses to be more informative (elastic/x-pack-elasticsearch#1598)
When an error response contains multiple layers of errors, Kibana displays
the one labelled root_cause.  The definition of root_cause is the most
deeply nested ElasticsearchException.  Therefore, it is of great benefit to
the UI if our config validation returns the actual problem in an
ElasticsearchException rather than an IllegalArgumentException.

This commit also adds an extra validation check to catch the case of a
single job config containing fields x.y as well as x earlier.  Previously
this was caught when we tried to create results mappings, and was
accompanied by an error suggesting that using a dedicated results index
would help, when clearly it won't for a clash in a single job config.

Fixes elastic/x-pack-kibana#1387
Fixes elastic/prelert-legacy#349

Original commit: elastic/x-pack-elasticsearch@7d1b7def6c
2017-05-31 14:42:10 +01:00
David Roberts ef25e2b604 [ML] When deleting a filter refresh the index immediately (elastic/x-pack-elasticsearch#1587)
Otherwise it's possible that the get_filter endpoint can return a filter that's been
deleted. Although this is the behaviour of the search API, specific metadata
management APIs should provide better guarantees.

Original commit: elastic/x-pack-elasticsearch@818495f176
2017-05-31 14:36:43 +01:00
markharwood 518f8a9120 Graph - reinstate correct Graph REST endpoint implementationof the form `_xpack/graph/_explore`. (elastic/x-pack-elasticsearch#1589)
Previous versions accidentally introduced an unconventional format for x-pack plugins.

relates elastic/x-pack-elasticsearch#1570

Original commit: elastic/x-pack-elasticsearch@38c42ae150
2017-05-31 14:17:34 +01:00
David Kyle 1759f70ceb [ML] Fix bwc streaming close job requests to v5.4 nodes (elastic/x-pack-elasticsearch#1586)
Original commit: elastic/x-pack-elasticsearch@0f02c8ddde
2017-05-31 10:28:16 +01:00
David Pilato 39cabad4d0 Fix bad link for Nexus OSS
Related to elastic/x-pack-elasticsearch#1301

Original commit: elastic/x-pack-elasticsearch@2273e8ba24
2017-05-31 10:36:09 +02:00
Suyog Rao e7b492e450 [Logstash] Add new reserved role for managing LS configs (elastic/x-pack-elasticsearch#1531)
Adds a new reserved role called `logstash_admin` that provides access
to `.logstash-*` index for managing configurations.

Will add documentation in old x-pack dir

relates elastic/x-pack-elasticsearch#609

Original commit: elastic/x-pack-elasticsearch@48c40e01f8
2017-05-30 15:04:16 -07:00
Nik Everett 503717b915 Build: allow building snapshot of release branches (elastic/x-pack-elasticsearch#1582)
This allows us to build both 5.5.0-SNAPSHOT and 5.4.1-SNAPSHOT
artifacts for backwards compatibility testing. It is a port of
elastic/elasticsearch:24870 to x-pack and will be super useful
when elastic/elasticsearch:24846 is ported to x-pack.

Original commit: elastic/x-pack-elasticsearch@0ea443f488
2017-05-30 09:04:27 -04:00
David Roberts 374e54233d [TEST] Ensure a well-defined cleanup order for YAML tests (elastic/x-pack-elasticsearch#1585)
Previously there were two @After methods in the XPackRestIT class, and
there is no guarantee about the order in which these run.  This commit
replaces these with a single @After method that calls the cleanup methods
in a well-defined order.

Original commit: elastic/x-pack-elasticsearch@d3ab366591
2017-05-30 13:19:06 +01:00
Tim Vernum e177f79aa3 Support wildcards in has_privileges API (elastic/x-pack-elasticsearch#1454)
The has_privileges API now supports wildcards.
The semantics are that the user must have a superset of the wildcard being checked.

---------------------
Role | Check | Result
---------------------
*    | foo*  | true
f*   | foo*  | true
foo* | foo*  | true
foo* | foo?  | true
foo? | foo?  | true
foo? | foo*  | false
foo  | foo*  | false

Original commit: elastic/x-pack-elasticsearch@817550db17
2017-05-30 13:40:29 +10:00
Tim Vernum da40720ef0 Rename LDAP setting meta_data to metadata (elastic/x-pack-elasticsearch#1455)
We don't hyphenate metadata anywhere else.
Also added tests for the LdapMetaDataResolver as they were completely absent.

Original commit: elastic/x-pack-elasticsearch@eec647ba93
2017-05-30 13:38:40 +10:00
Tanguy Leroux e977bdbf1f [Test] @AwaitsFix on BootStrapTests.testMixedTriggeredWatchLoading
It seems that it's going to be fixed by elastic/x-pack-elasticsearch#1544.

Original commit: elastic/x-pack-elasticsearch@4dce689b5b
2017-05-29 14:10:16 +02:00
David Pilato 7f0bc5dff4 Add documentation about configuring an enterprise Nexus instance (elastic/x-pack-elasticsearch#1301)
Original commit: elastic/x-pack-elasticsearch@cb10936aa7
2017-05-29 11:21:32 +02:00
Nik Everett 4e39bbb84a Rename some version constants to handle changes in core (elastic/x-pack-elasticsearch#1575)
Handle core renaming some constants.

Original commit: elastic/x-pack-elasticsearch@6db55e0225
2017-05-26 18:36:48 -04:00
Nik Everett 592af606f8 Build: fix the rolling restarts for real
I had the numbers backwards. 🤦

Original commit: elastic/x-pack-elasticsearch@8f2d21db52
2017-05-26 17:31:50 -04:00
Nik Everett 928589bd89 Build: Use correct data node in test
We were getting lock errors because I hand ported something and made
a copy and paste error.

Original commit: elastic/x-pack-elasticsearch@9896ba4cc7
2017-05-26 15:20:44 -04:00
Nik Everett 43928708fa Build: handle core's change to data directory
Core changed how the data directory is configured so we have to
conform to the new way.

Original commit: elastic/x-pack-elasticsearch@360e2fea1a
2017-05-26 13:32:39 -04:00
Nik Everett b215c66778 Add tests for wire compatibility when system key is enabled (elastic/x-pack-elasticsearch#1557)
Reworks the rolling restart tests so they'd have caugh an
incompatibility in the wire protocol that we say between 5.4.0 and
5.4.1.

Original commit: elastic/x-pack-elasticsearch@f5e69cf58e
2017-05-26 12:17:00 -04:00
David Roberts cc96580cd6 [ML] Remove record_count from bucket results (elastic/x-pack-elasticsearch#1568)
relates elastic/x-pack-elasticsearch#1564

Original commit: elastic/x-pack-elasticsearch@0caff1a735
2017-05-26 16:57:40 +01:00
David Kyle b284fc3c91 [Ml] Read v5.4 datacounts (elastic/x-pack-elasticsearch#1565)
* Read v54 datacounts

* Rename method legacyDocumentId -> v54DocumentId

Original commit: elastic/x-pack-elasticsearch@7dd297c287
2017-05-26 16:29:21 +01:00
Dimitris Athanasiou 8dc50990a3 [ML] Enable single_type on ML indices (elastic/x-pack-elasticsearch#1566)
Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@7e11ed2774
2017-05-26 15:51:42 +01:00
Dimitrios Athanasiou 8ec9427196 [TEST] Fix YAML tests for single type and new id changes
Original commit: elastic/x-pack-elasticsearch@a3b2bd6643
2017-05-26 15:43:15 +01:00
Dimitrios Athanasiou 9ed06ba15e [ML] Remove duplicate tests from MlJobIT
This deletes tests getting results from MlJobIT since
such tests already exist in a form that is simpler to
understand and maintain in the YAML suite.

Original commit: elastic/x-pack-elasticsearch@b708e24877
2017-05-26 13:44:45 +01:00
Alexander Reelsen e9fed20d96 Watcher: Use execute watch API in tests to prevent wrong watch history query
This test failed, because there could have been other results in the
watch history. However by switching to the execute watch API, there
is no need to query the watch history at all. This also removes a 5
second wait time in the test.

relates elastic/x-pack-elasticsearch#1549

Original commit: elastic/x-pack-elasticsearch@692779521e
2017-05-26 14:27:32 +02:00
Dimitris Athanasiou 156059e065 [ML] Add missing mappings on job open (elastic/x-pack-elasticsearch#1563)
As we now have single mappings for a doc type,
we need to add these mappings if they are missing
from existing indices that were created in 5.4.

Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@1693807b6e
2017-05-26 13:13:21 +01:00
David Roberts fffe424625 [ML] Switch state to use _type "doc" (elastic/x-pack-elasticsearch#1552)
This commit means that newly created ML state indices will have a single
type named "doc", and newly persisted state documents will have type
"doc" too.

Retrieving state is only supported for type "doc".

When deleting state, documents with the old types are deleted in addition
to those with type "doc".  This means jobs created by the beta can be fully
deleted.

Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@29c07d40f1
2017-05-26 10:51:29 +01:00
Ryan Ernst bb71839b85 Build: Switch ml snapshot dependency to a local project (elastic/x-pack-elasticsearch#1559)
This commit adds an internal project call ml-cpp-snapshot which when
built will pull the ml cpp zip file from the prelert bucket. The GET
request has retries added to handle the dynamic aws creds eventual
consistency.

Original commit: elastic/x-pack-elasticsearch@1bba7d0f08
2017-05-26 01:15:12 -07:00
Ryan Ernst 4ecd1e5d50 Fix mock painless to use mock compiled script for generating
search/executable scripts

Original commit: elastic/x-pack-elasticsearch@362432664b
2017-05-26 00:35:36 -07:00
Ryan Ernst d9816fac58 Update uses of script factory types to new names (elastic/x-pack-elasticsearch#1560)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24897

Original commit: elastic/x-pack-elasticsearch@d61f4e1da2
2017-05-26 00:03:00 -07:00
Ryan Ernst 77f441b1a0 Update signature for getScriptEngine method of script plugins
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24896

Original commit: elastic/x-pack-elasticsearch@6b06f5e758
2017-05-25 16:56:00 -07:00
Ryan Ernst d3b3fe783d Use new context constants for SearchScript and ExecutableScript (elastic/x-pack-elasticsearch#1550)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24883.

Original commit: elastic/x-pack-elasticsearch@9e612ec222
2017-05-25 12:18:55 -07:00
Ryan Ernst 404ba010ad Test: Fix ci script to use pr author's fork when looking for source branch (elastic/x-pack-elasticsearch#1548)
Original commit: elastic/x-pack-elasticsearch@faf72043be
2017-05-25 12:03:40 -07:00
Jay Modi f5e86cabaf Move PkiRealm checks for SSL and client authentication to a bootstrap check (elastic/x-pack-elasticsearch#1442)
This commit cleans up the check for SSL with client authentication when a PKI realm is enabled by
moving it from the realm to a actual bootstrap check.

A bug was found during this cleanup in the check for transport profiles and that is also fixed in
this commit.

relates elastic/x-pack-elasticsearch#420

Original commit: elastic/x-pack-elasticsearch@3aa6a3edc0
2017-05-25 12:58:45 -06:00
Dimitris Athanasiou 1e86f55746 [ML] Fix search that fetches results for renormalization (elastic/x-pack-elasticsearch#1556)
The commit that converted the results index into single type
broke the search for fetching results for renormalization.
This commit fixes that.

Original commit: elastic/x-pack-elasticsearch@1ca7517adc
2017-05-25 17:54:13 +01:00