Commit Graph

5759 Commits

Author SHA1 Message Date
David Kyle 1759f70ceb [ML] Fix bwc streaming close job requests to v5.4 nodes (elastic/x-pack-elasticsearch#1586)
Original commit: elastic/x-pack-elasticsearch@0f02c8ddde
2017-05-31 10:28:16 +01:00
David Pilato 39cabad4d0 Fix bad link for Nexus OSS
Related to elastic/x-pack-elasticsearch#1301

Original commit: elastic/x-pack-elasticsearch@2273e8ba24
2017-05-31 10:36:09 +02:00
Suyog Rao e7b492e450 [Logstash] Add new reserved role for managing LS configs (elastic/x-pack-elasticsearch#1531)
Adds a new reserved role called `logstash_admin` that provides access
to `.logstash-*` index for managing configurations.

Will add documentation in old x-pack dir

relates elastic/x-pack-elasticsearch#609

Original commit: elastic/x-pack-elasticsearch@48c40e01f8
2017-05-30 15:04:16 -07:00
Nik Everett 503717b915 Build: allow building snapshot of release branches (elastic/x-pack-elasticsearch#1582)
This allows us to build both 5.5.0-SNAPSHOT and 5.4.1-SNAPSHOT
artifacts for backwards compatibility testing. It is a port of
elastic/elasticsearch:24870 to x-pack and will be super useful
when elastic/elasticsearch:24846 is ported to x-pack.

Original commit: elastic/x-pack-elasticsearch@0ea443f488
2017-05-30 09:04:27 -04:00
David Roberts 374e54233d [TEST] Ensure a well-defined cleanup order for YAML tests (elastic/x-pack-elasticsearch#1585)
Previously there were two @After methods in the XPackRestIT class, and
there is no guarantee about the order in which these run.  This commit
replaces these with a single @After method that calls the cleanup methods
in a well-defined order.

Original commit: elastic/x-pack-elasticsearch@d3ab366591
2017-05-30 13:19:06 +01:00
Tim Vernum e177f79aa3 Support wildcards in has_privileges API (elastic/x-pack-elasticsearch#1454)
The has_privileges API now supports wildcards.
The semantics are that the user must have a superset of the wildcard being checked.

---------------------
Role | Check | Result
---------------------
*    | foo*  | true
f*   | foo*  | true
foo* | foo*  | true
foo* | foo?  | true
foo? | foo?  | true
foo? | foo*  | false
foo  | foo*  | false

Original commit: elastic/x-pack-elasticsearch@817550db17
2017-05-30 13:40:29 +10:00
Tim Vernum da40720ef0 Rename LDAP setting meta_data to metadata (elastic/x-pack-elasticsearch#1455)
We don't hyphenate metadata anywhere else.
Also added tests for the LdapMetaDataResolver as they were completely absent.

Original commit: elastic/x-pack-elasticsearch@eec647ba93
2017-05-30 13:38:40 +10:00
Tanguy Leroux e977bdbf1f [Test] @AwaitsFix on BootStrapTests.testMixedTriggeredWatchLoading
It seems that it's going to be fixed by elastic/x-pack-elasticsearch#1544.

Original commit: elastic/x-pack-elasticsearch@4dce689b5b
2017-05-29 14:10:16 +02:00
David Pilato 7f0bc5dff4 Add documentation about configuring an enterprise Nexus instance (elastic/x-pack-elasticsearch#1301)
Original commit: elastic/x-pack-elasticsearch@cb10936aa7
2017-05-29 11:21:32 +02:00
Nik Everett 4e39bbb84a Rename some version constants to handle changes in core (elastic/x-pack-elasticsearch#1575)
Handle core renaming some constants.

Original commit: elastic/x-pack-elasticsearch@6db55e0225
2017-05-26 18:36:48 -04:00
Nik Everett 592af606f8 Build: fix the rolling restarts for real
I had the numbers backwards. 🤦

Original commit: elastic/x-pack-elasticsearch@8f2d21db52
2017-05-26 17:31:50 -04:00
Nik Everett 928589bd89 Build: Use correct data node in test
We were getting lock errors because I hand ported something and made
a copy and paste error.

Original commit: elastic/x-pack-elasticsearch@9896ba4cc7
2017-05-26 15:20:44 -04:00
Nik Everett 43928708fa Build: handle core's change to data directory
Core changed how the data directory is configured so we have to
conform to the new way.

Original commit: elastic/x-pack-elasticsearch@360e2fea1a
2017-05-26 13:32:39 -04:00
Nik Everett b215c66778 Add tests for wire compatibility when system key is enabled (elastic/x-pack-elasticsearch#1557)
Reworks the rolling restart tests so they'd have caugh an
incompatibility in the wire protocol that we say between 5.4.0 and
5.4.1.

Original commit: elastic/x-pack-elasticsearch@f5e69cf58e
2017-05-26 12:17:00 -04:00
David Roberts cc96580cd6 [ML] Remove record_count from bucket results (elastic/x-pack-elasticsearch#1568)
relates elastic/x-pack-elasticsearch#1564

Original commit: elastic/x-pack-elasticsearch@0caff1a735
2017-05-26 16:57:40 +01:00
David Kyle b284fc3c91 [Ml] Read v5.4 datacounts (elastic/x-pack-elasticsearch#1565)
* Read v54 datacounts

* Rename method legacyDocumentId -> v54DocumentId

Original commit: elastic/x-pack-elasticsearch@7dd297c287
2017-05-26 16:29:21 +01:00
Dimitris Athanasiou 8dc50990a3 [ML] Enable single_type on ML indices (elastic/x-pack-elasticsearch#1566)
Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@7e11ed2774
2017-05-26 15:51:42 +01:00
Dimitrios Athanasiou 8ec9427196 [TEST] Fix YAML tests for single type and new id changes
Original commit: elastic/x-pack-elasticsearch@a3b2bd6643
2017-05-26 15:43:15 +01:00
Dimitrios Athanasiou 9ed06ba15e [ML] Remove duplicate tests from MlJobIT
This deletes tests getting results from MlJobIT since
such tests already exist in a form that is simpler to
understand and maintain in the YAML suite.

Original commit: elastic/x-pack-elasticsearch@b708e24877
2017-05-26 13:44:45 +01:00
Alexander Reelsen e9fed20d96 Watcher: Use execute watch API in tests to prevent wrong watch history query
This test failed, because there could have been other results in the
watch history. However by switching to the execute watch API, there
is no need to query the watch history at all. This also removes a 5
second wait time in the test.

relates elastic/x-pack-elasticsearch#1549

Original commit: elastic/x-pack-elasticsearch@692779521e
2017-05-26 14:27:32 +02:00
Dimitris Athanasiou 156059e065 [ML] Add missing mappings on job open (elastic/x-pack-elasticsearch#1563)
As we now have single mappings for a doc type,
we need to add these mappings if they are missing
from existing indices that were created in 5.4.

Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@1693807b6e
2017-05-26 13:13:21 +01:00
David Roberts fffe424625 [ML] Switch state to use _type "doc" (elastic/x-pack-elasticsearch#1552)
This commit means that newly created ML state indices will have a single
type named "doc", and newly persisted state documents will have type
"doc" too.

Retrieving state is only supported for type "doc".

When deleting state, documents with the old types are deleted in addition
to those with type "doc".  This means jobs created by the beta can be fully
deleted.

Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@29c07d40f1
2017-05-26 10:51:29 +01:00
Ryan Ernst bb71839b85 Build: Switch ml snapshot dependency to a local project (elastic/x-pack-elasticsearch#1559)
This commit adds an internal project call ml-cpp-snapshot which when
built will pull the ml cpp zip file from the prelert bucket. The GET
request has retries added to handle the dynamic aws creds eventual
consistency.

Original commit: elastic/x-pack-elasticsearch@1bba7d0f08
2017-05-26 01:15:12 -07:00
Ryan Ernst 4ecd1e5d50 Fix mock painless to use mock compiled script for generating
search/executable scripts

Original commit: elastic/x-pack-elasticsearch@362432664b
2017-05-26 00:35:36 -07:00
Ryan Ernst d9816fac58 Update uses of script factory types to new names (elastic/x-pack-elasticsearch#1560)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24897

Original commit: elastic/x-pack-elasticsearch@d61f4e1da2
2017-05-26 00:03:00 -07:00
Ryan Ernst 77f441b1a0 Update signature for getScriptEngine method of script plugins
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24896

Original commit: elastic/x-pack-elasticsearch@6b06f5e758
2017-05-25 16:56:00 -07:00
Ryan Ernst d3b3fe783d Use new context constants for SearchScript and ExecutableScript (elastic/x-pack-elasticsearch#1550)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24883.

Original commit: elastic/x-pack-elasticsearch@9e612ec222
2017-05-25 12:18:55 -07:00
Ryan Ernst 404ba010ad Test: Fix ci script to use pr author's fork when looking for source branch (elastic/x-pack-elasticsearch#1548)
Original commit: elastic/x-pack-elasticsearch@faf72043be
2017-05-25 12:03:40 -07:00
Jay Modi f5e86cabaf Move PkiRealm checks for SSL and client authentication to a bootstrap check (elastic/x-pack-elasticsearch#1442)
This commit cleans up the check for SSL with client authentication when a PKI realm is enabled by
moving it from the realm to a actual bootstrap check.

A bug was found during this cleanup in the check for transport profiles and that is also fixed in
this commit.

relates elastic/x-pack-elasticsearch#420

Original commit: elastic/x-pack-elasticsearch@3aa6a3edc0
2017-05-25 12:58:45 -06:00
Dimitris Athanasiou 1e86f55746 [ML] Fix search that fetches results for renormalization (elastic/x-pack-elasticsearch#1556)
The commit that converted the results index into single type
broke the search for fetching results for renormalization.
This commit fixes that.

Original commit: elastic/x-pack-elasticsearch@1ca7517adc
2017-05-25 17:54:13 +01:00
Dimitris Athanasiou 9b655ce6f1 [ML] Improve logging while removing expired data (elastic/x-pack-elasticsearch#1554)
relates elastic/x-pack-elasticsearch#1286

Original commit: elastic/x-pack-elasticsearch@4f938fa14b
2017-05-25 14:40:09 +01:00
Dimitris Athanasiou 779e6f6dba [ML] Handle requests with source (elastic/x-pack-elasticsearch#1553)
REST endpoints that support GET and POST need
to also support source parsing. As these
endpoints can accept a body but some clients
do not allow doing a GET with a request body,
elasticsearch has support for parsing via a
source URI parameter. This commit adds source
handling to all such endpoints.

relates elastic/x-pack-elasticsearch#1204

Original commit: elastic/x-pack-elasticsearch@3949ea31fe
2017-05-25 11:57:16 +01:00
David Kyle ce25e1f4f3 [ML] Don’t wait on flush ack if results parser has failed (elastic/x-pack-elasticsearch#1540)
Original commit: elastic/x-pack-elasticsearch@f1a82ae315
2017-05-25 10:52:44 +01:00
David Kyle 6befa83337 [ML] Reinstate DatafeedJobsIT::testRealtime_multipleStopCalls test (elastic/x-pack-elasticsearch#1542)
* Handle exception in action

Original commit: elastic/x-pack-elasticsearch@2c2f28115f
2017-05-25 10:02:14 +01:00
David Roberts 1bfc864193 [ML] Normalize records and buckets separately (elastic/x-pack-elasticsearch#1524)
Previously we used to normalize records with their buckets.  This required
nested scrolls: an outer scroll over buckets, then a nested scroll for
records in each bucket.  This was fragile.

The new approach is to simply scroll first through buckets, then through
records.  This is made possible because we no longer store max_record_score
on buckets nor bucket anomaly_score on records.

While making these changes I noticed that the PerPartitionMaxProbabilities
class was redundant (because it was storing max_record_score in the case of
per-partition normalization), so I removed it.  I also removed a redundant
Map from the Bucket class and fixed its equals() and hashCode() methods.

relates elastic/x-pack-elasticsearch#1115

Original commit: elastic/x-pack-elasticsearch@efbee63573
2017-05-25 08:50:56 +01:00
Ryan Ernst aae7cf0b0f Update test script engines to use generic compile method (elastic/x-pack-elasticsearch#1546)
This is the xpack side of https://github.com/elastic/elasticsearch/pull/24873

Original commit: elastic/x-pack-elasticsearch@1779afa6bc
2017-05-24 20:06:51 -07:00
Ryan Ernst 3b58334efb Refactor script context generic types (elastic/x-pack-elasticsearch#1547)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24877

Original commit: elastic/x-pack-elasticsearch@4f3a1a27e6
2017-05-24 19:20:59 -07:00
Alexander Reelsen 463c337d62 Tests: Ensure HTTPClient does not follow redirects after post
Relates elastic/x-pack-elasticsearch#470

Original commit: elastic/x-pack-elasticsearch@5333a65c0e
2017-05-24 23:46:14 +02:00
Ryan Ernst 9a7c28786a Scripting: Use type aware script contexts (elastic/x-pack-elasticsearch#1538)
This creates two different script contexts for watcher, one which may be
used for SearchScript, and another for ExecutableScript.
This is the xpack side of elastic/elasticsearch#24868.

Original commit: elastic/x-pack-elasticsearch@9ae3d45fed
2017-05-24 14:29:25 -07:00
Dimitris Athanasiou d7e528f7f7 [ML] Reject put datafeed for job that is marked as deleted (elastic/x-pack-elasticsearch#1537)
Deleting a job issues 2 cluster state updates.
The first marks the job as deleted.
The second actually removes the job.
Both check that there is no datafeed referring to the job.
If a put datafeed request arrives between those 2 cluster
state updates, the datafeed gets created and the final
job cluster state update fails. This means we end up with
both the job and the datafeed, but the job's results and
state have been deleted.

This commit changes the behaviour so that the put
datafeed request fails for a job that is marked as deleted
as this scenario is avoiding partially executing an action.

relates elastic/x-pack-elasticsearch#1510 

Original commit: elastic/x-pack-elasticsearch@76fa0f0b1a
2017-05-24 18:34:03 +01:00
Alexander Reelsen ccf3fa2579 Watcher: Muting failing BootStrapTests.testTriggeredWatchLoading test
Relates elastic/x-pack-elasticsearch#1309

Original commit: elastic/x-pack-elasticsearch@ed03afc4d1
2017-05-24 14:55:39 +02:00
Dimitris Athanasiou 71fe599592 [ML] Single doc type for results index (elastic/x-pack-elasticsearch#1528)
This commit changes all results to use the single doc type.
All searches were adjusted to work without the need to specify
type which ensures BWC with 5.4 results.

Additional work is needed to put the new doc mapping in indices
created with 5.4 but it will be done in separate PR.

Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@041c88ac2d
2017-05-24 13:24:32 +01:00
Alexander Reelsen 3d057991e0 Tests: Various watcher test improvements
* Reduced a longish timeout to a shorter one, as a watch should be
  executed in a HTTP test.
* Ensured that the TimeThrottleIntegration tests only query for own
  watches in the watch history, also use random names for watch ids
* HipChatServiceTests configured deprecated logging package, so it was
  not possible to follow the HTTP calls to the hipchat service endpoint.

relates elastic/x-pack-elasticsearch#1514
Relates elastic/x-pack-elasticsearch#1515

Original commit: elastic/x-pack-elasticsearch@adb492e4e9
2017-05-24 14:08:02 +02:00
David Kyle caad9035fd Mark failing test AwaitsFix
Original commit: elastic/x-pack-elasticsearch@c117ef9947
2017-05-24 12:25:34 +01:00
David Kyle b083062689 [ML] Add document type to ID (elastic/x-pack-elasticsearch#1525)
* Add document type to ID

* Delete v5.4 quantiles

Original commit: elastic/x-pack-elasticsearch@d1f383b972
2017-05-24 11:43:25 +01:00
David Kyle fc2d1266f4 Fix compilation
Original commit: elastic/x-pack-elasticsearch@6d34ac5a0c
2017-05-24 10:04:08 +01:00
David Kyle d9882be292 [ML] Use unique and predictable IDs for result docs (elastic/x-pack-elasticsearch#1521)
* Remove sequenceNum from anomaly records and influencers

* Generate unqiue IDs without sequence numbers

* Remove more instances of sequence_num

* Handle parsing sequnce_num from v5.4

Original commit: elastic/x-pack-elasticsearch@e60b206daf
2017-05-24 09:59:17 +01:00
Jason Tedor 284bf2512a Fix rolling upgrade test dependency
The rolling upgrade nodes need a keystore for SSL configuration but
there was no dependency on the task that copies the keystore into the
output directory for the nodes to pick up as an extra configuration
file. This commit addresses this by adding such a dependency. To do
this, we need to break the dependency of the keystore copy task on the
REST spec copy task; this is not an issue since the dependency was for
convenience of ordering the task and not actually needed.

Original commit: elastic/x-pack-elasticsearch@fddbc06e9f
2017-05-23 21:44:59 -04:00
Lisa Cawley 386ac7345c [DOCS] Add details about using aggregations with machine learning (elastic/x-pack-elasticsearch#1446)
* [DOCS] Add ML aggregations configuration scenario

* [DOCS] Refine ML configuration page

* [DOCS] Add ML aggregation details

* [DOCS] Add links to aggregations in Configuring ML

* [DOCS] Address feedback about ML aggregations

Original commit: elastic/x-pack-elasticsearch@8474144093
2017-05-23 14:34:21 -07:00
Ryan Ernst b664e66a0d Test: Simplify watch stats test script engine (elastic/x-pack-elasticsearch#1520)
This commit simplifies the WatchStatsTests to use a MockScriptPlugin.
The latch script engine previously depended on a static instance of the
engine to contain the latches. These are now moved to statics of the
test class itself.

Original commit: elastic/x-pack-elasticsearch@4170cd1bd3
2017-05-23 12:23:05 -07:00