Commit Graph

5778 Commits

Author SHA1 Message Date
Dimitris Athanasiou 779e6f6dba [ML] Handle requests with source (elastic/x-pack-elasticsearch#1553)
REST endpoints that support GET and POST need
to also support source parsing. As these
endpoints can accept a body but some clients
do not allow doing a GET with a request body,
elasticsearch has support for parsing via a
source URI parameter. This commit adds source
handling to all such endpoints.

relates elastic/x-pack-elasticsearch#1204

Original commit: elastic/x-pack-elasticsearch@3949ea31fe
2017-05-25 11:57:16 +01:00
David Kyle ce25e1f4f3 [ML] Don’t wait on flush ack if results parser has failed (elastic/x-pack-elasticsearch#1540)
Original commit: elastic/x-pack-elasticsearch@f1a82ae315
2017-05-25 10:52:44 +01:00
David Kyle 6befa83337 [ML] Reinstate DatafeedJobsIT::testRealtime_multipleStopCalls test (elastic/x-pack-elasticsearch#1542)
* Handle exception in action

Original commit: elastic/x-pack-elasticsearch@2c2f28115f
2017-05-25 10:02:14 +01:00
David Roberts 1bfc864193 [ML] Normalize records and buckets separately (elastic/x-pack-elasticsearch#1524)
Previously we used to normalize records with their buckets.  This required
nested scrolls: an outer scroll over buckets, then a nested scroll for
records in each bucket.  This was fragile.

The new approach is to simply scroll first through buckets, then through
records.  This is made possible because we no longer store max_record_score
on buckets nor bucket anomaly_score on records.

While making these changes I noticed that the PerPartitionMaxProbabilities
class was redundant (because it was storing max_record_score in the case of
per-partition normalization), so I removed it.  I also removed a redundant
Map from the Bucket class and fixed its equals() and hashCode() methods.

relates elastic/x-pack-elasticsearch#1115

Original commit: elastic/x-pack-elasticsearch@efbee63573
2017-05-25 08:50:56 +01:00
Ryan Ernst aae7cf0b0f Update test script engines to use generic compile method (elastic/x-pack-elasticsearch#1546)
This is the xpack side of https://github.com/elastic/elasticsearch/pull/24873

Original commit: elastic/x-pack-elasticsearch@1779afa6bc
2017-05-24 20:06:51 -07:00
Ryan Ernst 3b58334efb Refactor script context generic types (elastic/x-pack-elasticsearch#1547)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24877

Original commit: elastic/x-pack-elasticsearch@4f3a1a27e6
2017-05-24 19:20:59 -07:00
Alexander Reelsen 463c337d62 Tests: Ensure HTTPClient does not follow redirects after post
Relates elastic/x-pack-elasticsearch#470

Original commit: elastic/x-pack-elasticsearch@5333a65c0e
2017-05-24 23:46:14 +02:00
Ryan Ernst 9a7c28786a Scripting: Use type aware script contexts (elastic/x-pack-elasticsearch#1538)
This creates two different script contexts for watcher, one which may be
used for SearchScript, and another for ExecutableScript.
This is the xpack side of elastic/elasticsearch#24868.

Original commit: elastic/x-pack-elasticsearch@9ae3d45fed
2017-05-24 14:29:25 -07:00
Dimitris Athanasiou d7e528f7f7 [ML] Reject put datafeed for job that is marked as deleted (elastic/x-pack-elasticsearch#1537)
Deleting a job issues 2 cluster state updates.
The first marks the job as deleted.
The second actually removes the job.
Both check that there is no datafeed referring to the job.
If a put datafeed request arrives between those 2 cluster
state updates, the datafeed gets created and the final
job cluster state update fails. This means we end up with
both the job and the datafeed, but the job's results and
state have been deleted.

This commit changes the behaviour so that the put
datafeed request fails for a job that is marked as deleted
as this scenario is avoiding partially executing an action.

relates elastic/x-pack-elasticsearch#1510 

Original commit: elastic/x-pack-elasticsearch@76fa0f0b1a
2017-05-24 18:34:03 +01:00
Alexander Reelsen ccf3fa2579 Watcher: Muting failing BootStrapTests.testTriggeredWatchLoading test
Relates elastic/x-pack-elasticsearch#1309

Original commit: elastic/x-pack-elasticsearch@ed03afc4d1
2017-05-24 14:55:39 +02:00
Dimitris Athanasiou 71fe599592 [ML] Single doc type for results index (elastic/x-pack-elasticsearch#1528)
This commit changes all results to use the single doc type.
All searches were adjusted to work without the need to specify
type which ensures BWC with 5.4 results.

Additional work is needed to put the new doc mapping in indices
created with 5.4 but it will be done in separate PR.

Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@041c88ac2d
2017-05-24 13:24:32 +01:00
Alexander Reelsen 3d057991e0 Tests: Various watcher test improvements
* Reduced a longish timeout to a shorter one, as a watch should be
  executed in a HTTP test.
* Ensured that the TimeThrottleIntegration tests only query for own
  watches in the watch history, also use random names for watch ids
* HipChatServiceTests configured deprecated logging package, so it was
  not possible to follow the HTTP calls to the hipchat service endpoint.

relates elastic/x-pack-elasticsearch#1514
Relates elastic/x-pack-elasticsearch#1515

Original commit: elastic/x-pack-elasticsearch@adb492e4e9
2017-05-24 14:08:02 +02:00
David Kyle caad9035fd Mark failing test AwaitsFix
Original commit: elastic/x-pack-elasticsearch@c117ef9947
2017-05-24 12:25:34 +01:00
David Kyle b083062689 [ML] Add document type to ID (elastic/x-pack-elasticsearch#1525)
* Add document type to ID

* Delete v5.4 quantiles

Original commit: elastic/x-pack-elasticsearch@d1f383b972
2017-05-24 11:43:25 +01:00
David Kyle fc2d1266f4 Fix compilation
Original commit: elastic/x-pack-elasticsearch@6d34ac5a0c
2017-05-24 10:04:08 +01:00
David Kyle d9882be292 [ML] Use unique and predictable IDs for result docs (elastic/x-pack-elasticsearch#1521)
* Remove sequenceNum from anomaly records and influencers

* Generate unqiue IDs without sequence numbers

* Remove more instances of sequence_num

* Handle parsing sequnce_num from v5.4

Original commit: elastic/x-pack-elasticsearch@e60b206daf
2017-05-24 09:59:17 +01:00
Jason Tedor 284bf2512a Fix rolling upgrade test dependency
The rolling upgrade nodes need a keystore for SSL configuration but
there was no dependency on the task that copies the keystore into the
output directory for the nodes to pick up as an extra configuration
file. This commit addresses this by adding such a dependency. To do
this, we need to break the dependency of the keystore copy task on the
REST spec copy task; this is not an issue since the dependency was for
convenience of ordering the task and not actually needed.

Original commit: elastic/x-pack-elasticsearch@fddbc06e9f
2017-05-23 21:44:59 -04:00
Lisa Cawley 386ac7345c [DOCS] Add details about using aggregations with machine learning (elastic/x-pack-elasticsearch#1446)
* [DOCS] Add ML aggregations configuration scenario

* [DOCS] Refine ML configuration page

* [DOCS] Add ML aggregation details

* [DOCS] Add links to aggregations in Configuring ML

* [DOCS] Address feedback about ML aggregations

Original commit: elastic/x-pack-elasticsearch@8474144093
2017-05-23 14:34:21 -07:00
Ryan Ernst b664e66a0d Test: Simplify watch stats test script engine (elastic/x-pack-elasticsearch#1520)
This commit simplifies the WatchStatsTests to use a MockScriptPlugin.
The latch script engine previously depended on a static instance of the
engine to contain the latches. These are now moved to statics of the
test class itself.

Original commit: elastic/x-pack-elasticsearch@4170cd1bd3
2017-05-23 12:23:05 -07:00
Jason Tedor ed827970f1 Mark failing password migration test as skipped
A previous commit marked the wrong test as skipped, this commit marks
the correct issue.

Original commit: elastic/x-pack-elasticsearch@0723964206
2017-05-23 14:25:02 -04:00
Jason Tedor 3a151ac320 Mark failing password migration test as skipped
This test is failing reliably, so this commit marks it as skipped.

Original commit: elastic/x-pack-elasticsearch@15afd615d3
2017-05-23 14:13:57 -04:00
Jason Tedor dde2600181 Mark verify Kibana user works as skipped
This test fails reliably, so we mark it as skipped.

Original commit: elastic/x-pack-elasticsearch@4fc8a6883e
2017-05-23 13:35:44 -04:00
Jason Tedor 6ce8bd69ef Mark default password migration test as skipped
This test fails reliably, so this commit marks the test as awaits fix.

Original commit: elastic/x-pack-elasticsearch@8eb8b732ba
2017-05-23 13:35:44 -04:00
Clinton Gormley 844a97586e Updated docs for cross-cluster search (elastic/x-pack-elasticsearch#1478)
* Updated docs for cross-cluster search

Wildcards are now supported in both cluster alias and index name,
and the "view_index_metadata" privilege is automatically conferred
by the "read_cross_cluster" role.

* For now, best to allow users permissions to query all indices on all remote clusters

Original commit: elastic/x-pack-elasticsearch@31959f78f4
2017-05-23 19:18:03 +02:00
Jason Tedor 87ff7f0b52 Mark default password migration test as awaits fix
This commit marks the default password migration test as awaits fix as
the test is reliabily failling.

Original commit: elastic/x-pack-elasticsearch@4377480d98
2017-05-23 12:02:11 -04:00
Andrew Cholakian ae778cc66f Move logstash-states template into the main logstash template (elastic/x-pack-elasticsearch#1490)
This is a revert of elastic/x-pack-elasticsearch@1940d79d13.
and also moves the template to boot

Original commit: elastic/x-pack-elasticsearch@434183b0e9
2017-05-23 09:20:46 -05:00
Jason Tedor 275bfc0e51 Mark failing exporter template test as awaits fix
This test is failing for eleven days and needs to be looked at.

Original commit: elastic/x-pack-elasticsearch@b16b95f333
2017-05-23 09:48:21 -04:00
David Kyle ea0f3fe4a0 [ML] Wait for a stopping datafeed (elastic/x-pack-elasticsearch#1461)
* [ML] Wait for a stopping datafeed

* Fix compilation after rebase

* Address review comments

Original commit: elastic/x-pack-elasticsearch@2baed641e9
2017-05-23 14:31:19 +01:00
Ali Beyad ec175debd0 Enables security to work with index aliases (elastic/x-pack-elasticsearch#1496)
This commit enables security to work with an index named .security (as
it could before) OR an alias named .security that points to a concrete
index by a different name that has the security index.  This prepares
the ability to migrate from a 5.x to 6.x security index that allows
changing and re-indexing the underlying security index while maintaining
a .security alias that points to the underlying updated index.

relates elastic/x-pack-elasticsearch#1216

Original commit: elastic/x-pack-elasticsearch@9fee12e5a0
2017-05-23 09:10:06 -04:00
Alexander Reelsen 521b7a1940 Watcher: Support setting of markdown in attachments (elastic/x-pack-elasticsearch#1492)
Watcher: Support setting of markdown in attachments

This allows to set the `mrkdwn_in` array in slack attachments to specify which fields should contain markdown.
See https://api.slack.com/docs/message-formatting#message_formatting

This is mainly useful for the text and pretext fields in an attachment.

relates elastic/x-pack-elasticsearch#403

Original commit: elastic/x-pack-elasticsearch@ee4180e4bf
2017-05-23 13:33:00 +02:00
David Roberts e198845614 [ML] Avoid masses of log spam/stack traces on shutdown (elastic/x-pack-elasticsearch#1495)
Suppress many job/datafeed errors if a node is known to be shutting down. Also, ensure started datafeeds and open jobs don't end up stopped/failed due to errors as the shutdown progresses, as this would prevent them automatically relocating to a different node.

relates elastic/x-pack-elasticsearch#1114

Original commit: elastic/x-pack-elasticsearch@e56a7dbea1
2017-05-23 10:05:27 +01:00
Ali Beyad 958ee95f26 [TEST] mute watcher test failure that is being tracked in elastic/x-pack-elasticsearch#1517
Original commit: elastic/x-pack-elasticsearch@a28b7ce3a5
2017-05-22 22:01:31 -04:00
Ryan Ernst 68e0bc419a Use simplified ScriptContext (elastic/x-pack-elasticsearch#1502)
This is the xpack side of elastic/elasticsearch#24818

Original commit: elastic/x-pack-elasticsearch@0edbbab431
2017-05-22 13:11:23 -07:00
Ali Beyad f158be89b8 [TEST] fixes tests that attempt to delete documents from missing
indices, as that is now prohibited by elastic/elasticsearch#24518

relates elastic/x-pack-elasticsearch#1511

Original commit: elastic/x-pack-elasticsearch@026a516196
2017-05-22 13:16:53 -04:00
Ali Beyad e0a8881cfc [TEST] mute failing tests, which are tracked in elastic/x-pack-elasticsearch#1511
Original commit: elastic/x-pack-elasticsearch@e23dd847f3
2017-05-22 12:19:56 -04:00
Ryan Ernst 5a3c85cd57 Build: Add bwcTest target to ci script (elastic/x-pack-elasticsearch#1488)
This is to allow bwcTest to be run by ci. This commit also fixes the
existing packagingTest target, which was never updated after the xpack
repo split.

Original commit: elastic/x-pack-elasticsearch@f4696c1899
2017-05-22 08:53:36 -07:00
Dimitris Athanasiou aff8258398 [ML] Change result index searches to not use _type (elastic/x-pack-elasticsearch#1509)
Adjusts the searches for

- buckets
- categories
- model snapshots

to not use _type.

Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@8269609705
2017-05-22 14:44:39 +01:00
Hendrik Muhs edc299a532 move verification so that test is in order (elastic/x-pack-elasticsearch#1504)
improves test by taking order of statuses into account

Original commit: elastic/x-pack-elasticsearch@0d214714d3
2017-05-22 15:44:09 +02:00
David Kyle 1873624a18 Fix failing line length checks
Original commit: elastic/x-pack-elasticsearch@6a493a70a4
2017-05-22 12:39:51 +01:00
David Kyle 8e890d0365 Fix test
Original commit: elastic/x-pack-elasticsearch@cdf7950d3f
2017-05-22 12:36:15 +01:00
David Kyle 0425f58d80 [ML] Give test jobs meaningful names (elastic/x-pack-elasticsearch#1508)
Original commit: elastic/x-pack-elasticsearch@97bec3b1e9
2017-05-22 12:22:39 +01:00
David Roberts 5b2ef6e98e [ML] Ignore IndexNotFoundException when deleting quantiles (elastic/x-pack-elasticsearch#1507)
Relates elastic/elasticsearch#24518

Original commit: elastic/x-pack-elasticsearch@34ee52443c
2017-05-22 11:04:27 +01:00
Hendrik Muhs 527dcfd98d [ML] Make job closing robust against crashes in autodetect and other misbehavior (elastic/x-pack-elasticsearch#1480)
Set job to failed if autodetect manager fails closing, fix force closing of jobs that hang in closing 
state, set timeout when waiting for clusterstate update, disallow closing of failed jobs with normal 
close

relates elastic/x-pack-elasticsearch#1453 

Original commit: elastic/x-pack-elasticsearch@493cf85e22
2017-05-22 08:48:33 +02:00
Simon Willnauer 392e67851e Preserve aliases in index permissions (elastic/x-pack-elasticsearch#1501)
Aliases might be contained in requests that check index permissions
to disable caches etc. This commit preserves permissions for
aliases as well.

Original commit: elastic/x-pack-elasticsearch@233195aeba
2017-05-20 21:34:38 +02:00
Ryan Ernst 883f5d8a7a Remove overrides of isInlineScriptEnabled (elastic/x-pack-elasticsearch#1500)
This is the xpack side of elastic/elasticsearch#24815

Original commit: elastic/x-pack-elasticsearch@5aa314a49d
2017-05-20 12:01:34 -07:00
Lisa Cawley d18c48f820 [DOCS] Add info about configuring ML (elastic/x-pack-elasticsearch#1497)
Original commit: elastic/x-pack-elasticsearch@a9dfc239cc
2017-05-19 13:40:50 -07:00
Lisa Cawley d72a8f8026 [DOCS] Add ML geographic functions (elastic/x-pack-elasticsearch#1357)
* [DOCS] Add ML geographic functions

* [DOCS] Add script_fields info to ML geo functions

* [DOCS] Remove summary count from ML geographic functions

* [DOCS] Added example title to geographic functions

* [DOCS] Remove list from ML geographic functions

Original commit: elastic/x-pack-elasticsearch@a8e495657f
2017-05-19 11:09:39 -07:00
Lisa Cawley 27b0af7eae [DOCS] Modify ML analytical functions (elastic/x-pack-elasticsearch#1467)
* [DOCS] Modify ML analytical functions

* [DOCS] Fix ML function section titles

Original commit: elastic/x-pack-elasticsearch@f95ae012bb
2017-05-19 10:48:15 -07:00
David Roberts fa95474ab8 [BUILD] Change ordering of Java compilation and ML C++ notice extraction
May help to avoid problems with the speed of temporary AWS credentials
propagation (see elastic/x-pack-logstash#73)

Original commit: elastic/x-pack-elasticsearch@c78e00cda5
2017-05-19 13:36:40 +01:00
Tim Vernum b689159077 [TEST] Mock getVersion on channel for ServerTransportFilterTests
Original commit: elastic/x-pack-elasticsearch@0b48c21eb0
2017-05-19 10:15:49 +10:00