Commit Graph

6010 Commits

Author SHA1 Message Date
Alexander Reelsen 4346431156 Tests: Ensure watcher is started in REST tests (elastic/x-pack-elasticsearch#1702)
This adds a check in the REST tests to ensure that
watcher is started, and if not, tries to start watcher.

This eliminates test failures where watcher was not in
the correct state due to other tests stopping watcher.

Original commit: elastic/x-pack-elasticsearch@fc547d49b4
2017-06-14 08:34:09 +02:00
Ryan Ernst abe217ebc3 Remove uses of Strings.cleanPath (elastic/x-pack-elasticsearch#1709)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25209

Original commit: elastic/x-pack-elasticsearch@4f59f011e9
2017-06-13 21:10:02 -07:00
Lisa Cawley 9311d056ad [DOCS] Further doc build clarifications in README
Original commit: elastic/x-pack-elasticsearch@47393500b7
2017-06-13 14:55:45 -07:00
Lisa Cawley 4ab141d0d0 [DOCS] Add documentation build info to README
Original commit: elastic/x-pack-elasticsearch@b42ae5d8d4
2017-06-13 14:36:35 -07:00
Lisa Cawley dcc4fd28da [DOCS] Change standalone node to dedicated node (elastic/x-pack-elasticsearch#1708)
Original commit: elastic/x-pack-elasticsearch@41e13a2de1
2017-06-13 14:04:15 -07:00
Lisa Cawley f84ae92e51 [DOCS] Add X-Pack node settings for Elasticsearch Reference (elastic/x-pack-elasticsearch#1688)
* [DOCS] Add X-Pack node settings for Elasticsearch Reference

* [DOCS] Fix callout in node.asciidoc

Original commit: elastic/x-pack-elasticsearch@9c29441734
2017-06-13 12:13:29 -07:00
Suyog Rao fe72991c70 [Logstash] Add new component to x-pack to handle LS features (elastic/x-pack-elasticsearch#1530)
This commit adds a new Logstash component to x-pack to support the config management work. Currently, the functionality in this component is really simple; all it does is upload a new index template for `.logstash` index. This index stores the actual LS configuration.

On this template is bootstrapped in ES, Kibana can write user-created LS configs which adhere to the mapping defined here. In the future, we're looking into adding more functionality on the ES side to handle config documents, but for now, this is simple.

relates elastic/x-pack-elasticsearch#1499, relates elastic/x-pack-elasticsearch#1471

Original commit: elastic/x-pack-elasticsearch@d7cc8675f7
2017-06-13 10:30:30 -07:00
Dimitris Athanasiou f2e2ccae01 [ML] Extract parent field when job has text multi-field (elastic/x-pack-elasticsearch#1705)
In the case where a field is a text multi-field, it has
no doc values and it is not in source. Thus, the datafeed
will not be able to extract it.

However, it is possible to extract it by getting its parent
field instead. This commit implements the logic to look
in parent fields when the field in question is a text field.

Original commit: elastic/x-pack-elasticsearch@f116e89921
2017-06-13 18:00:24 +01:00
Clinton Gormley fff33e753a Corrected syntax for adding IDs to definition lists in built-in roles
Relates to elastic/x-pack-elasticsearch#5040

Original commit: elastic/x-pack-elasticsearch@693c4adfb5
2017-06-13 14:00:56 +02:00
Clinton Gormley 5a0ffb4b3a Fixed bad asciidoc link
relates elastic/x-pack-elasticsearch#5040

Original commit: elastic/x-pack-elasticsearch@feb5afbee4
2017-06-13 11:54:06 +02:00
Simon Willnauer 65bac10eed Fix API change in SecurityNetty4Transport
Original commit: elastic/x-pack-elasticsearch@f152fb1813
2017-06-13 10:15:26 +02:00
Alexander Reelsen b86c2e6e18 Watcher: Fix ack watch documentation (elastic/x-pack-elasticsearch#1695)
The ack watch documentation was exlcuded from the tests and
thus contained wrong information. This commit cleans it up.

relates elastic/x-pack-elasticsearch#1642

Original commit: elastic/x-pack-elasticsearch@81d69174df
2017-06-13 09:57:10 +02:00
Lisa Cawley 62ee1bc635 [DOCS] Add ML categorization of messages (elastic/x-pack-elasticsearch#1666)
* [DOCS] Add ML categorization of messages

* [DOCS] Describe ML categorization_examples_limit property

* [DOCS] Updated ML categorization of messages

* [DOCS] Add links to ML categorization

Original commit: elastic/x-pack-elasticsearch@6403f6ce84
2017-06-12 10:41:14 -07:00
Dimitrios Athanasiou 29811ea1d8 [ML] Write null instead of false for optional boolean that was removed
Original commit: elastic/x-pack-elasticsearch@43f5fc04bd
2017-06-12 17:35:25 +01:00
Dimitrios Athanasiou 1d33fb8b8b [TEST] Add now required mapping for data index in datafeed test
Original commit: elastic/x-pack-elasticsearch@67cf68c5a0
2017-06-12 15:50:16 +01:00
James Baiera 1d3921f581 Fixing vagrant build file issues with new vagrant support plugin
Original commit: elastic/x-pack-elasticsearch@dbb5d4a215
2017-06-12 10:03:30 -04:00
Dimitris Athanasiou 8eb62eac27 [ML] Automate detection of way to extract fields (elastic/x-pack-elasticsearch#1686)
In 5.4.x, the datafeed attempts to get all fields from
doc_values by default. It has a `_source` parameter which
when enabled changes the strategy to instead try to get
all fields from the source.

This has been the most common issue users have been
reporting as it means the datafeed will fail to fetch
any text fields by default.

This change uses the field capabilities API in order
to automatically detect whether a field is aggregatable.
It then extracts such fields from doc_values while the
rest are taken from source. The change also adds
validation to the start datafeed action so that if
fields are missing mappings or the time field is not
aggregatable we respond with an appropriate error.

relates elastic/x-pack-elasticsearch#1649

Original commit: elastic/x-pack-elasticsearch@76e2cc6cb2
2017-06-12 14:56:31 +01:00
Dimitris Athanasiou c9bbc17742 [ML] Make datafeed types optional (elastic/x-pack-elasticsearch#1690)
Specifying types for a datafeed should be optional
as no types is equal to searching through all types.

Original commit: elastic/x-pack-elasticsearch@f61ac01b45
2017-06-12 14:46:53 +01:00
Alexander Reelsen 27b5142de6 Watcher: Fix croneval tool for packaging (elastic/x-pack-elasticsearch#1689)
The croneval script used an old parameter to start up.
This commit removes this parameter, that is used, when a
package is used.

In addition a concrete vagrant test has been added.

relates elastic/x-pack-elasticsearch#1635

Original commit: elastic/x-pack-elasticsearch@ea7b8a08f4
2017-06-12 13:56:26 +02:00
Alexander Reelsen 83a5e022cb Watcher: Reduce logging noise when watcher might be stopped (elastic/x-pack-elasticsearch#1685)
Only log an entry when an actual stop is executed instead of
always logging.

Also added a reason to stop watcher to the methods, so that
debug logs will yield that information.

Original commit: elastic/x-pack-elasticsearch@8efaed0e9a
2017-06-12 10:24:16 +02:00
David Kyle 516696bbec [ML][TEST] Refresh after indexing docs
Original commit: elastic/x-pack-elasticsearch@b64ad1b2c6
2017-06-10 19:12:52 +01:00
Boaz Leskes b5ab68fac8 qa/upgrade_cluster/10_basic.yaml add shard level info on health failure
Original commit: elastic/x-pack-elasticsearch@46847ca262
2017-06-09 21:59:46 +02:00
Jason Tedor cb62d81bdc Fix compilation for CreateIndexResponse
The constructor for CreateIndexResponse changed to include the index
name. This commit adapts x-pack-elasticsearch to this change.

Original commit: elastic/x-pack-elasticsearch@b078d80cd9
2017-06-09 14:59:55 -04:00
Chris Earle c356074606 [Test] Rolling Upgrade should preserve templates (elastic/x-pack-elasticsearch#1687)
Real upgrades will have preserved their templates, so rolling upgrade tests should preserve them as well and internal services should be expected to replace them as needed.

Original commit: elastic/x-pack-elasticsearch@93a155951e
2017-06-09 13:41:22 -04:00
jaymode 9b3ee9f96c Remove security trace logging from rolling uprade tests
Original commit: elastic/x-pack-elasticsearch@b0ecb80816
2017-06-09 10:02:44 -06:00
jaymode 741bf85fbf Remove incorrect bwc branch logic from master
Commit elastic/x-pack-elasticsearch@b07aa78a7b was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement and project name.

Original commit: elastic/x-pack-elasticsearch@9a77269fa6
2017-06-09 10:02:43 -06:00
Alexander Reelsen 709ed7d50e Watcher: Only try to load triggered watches index, if it exists (elastic/x-pack-elasticsearch#1569)
This is mainly a commit to reduce noise in test logfiles when going
through them. When watcher shuts down and another node takes over, it
might try to start watcher again and tries to load triggered watches.
However the triggered watches index could be gone in the meantime due to
further shutdown. This results in logging a stack trace that the index
does not exist.

This commit checks the cluster state before trying to load triggered
watches to prevent an IndexNotFoundException in the logs.

Original commit: elastic/x-pack-elasticsearch@9f26d557d0
2017-06-09 17:51:23 +02:00
Alexander Reelsen d769ee0813 Watcher: Do not pause watcher if not needed (elastic/x-pack-elasticsearch#1681)
This introduces a check to only pause the execution of watcher
when there is no metadata but there was a shard on this node before
that inside of the ClusterStateListener.

This prevents repeated logging that watcher was paused even
though it was not necessary to call anything.

Original commit: elastic/x-pack-elasticsearch@8d3a829ffb
2017-06-09 17:50:35 +02:00
Alexander Reelsen 7f48337bf6 Watcher: Remove index template configuration via cluster settings (elastic/x-pack-elasticsearch#1680)
Watcher had an undocumented feature to configure the settings of any
index template via updating the cluster settings. Instead of changing
the template one could add a setting during runtime that overwrote
what is written in the index template.

As index template are created once and not overwritten, users should
just change the index template - or the concrete index settings like
number of replicas.

This feature was not exposed in our documentation at all.

Original commit: elastic/x-pack-elasticsearch@32e1769925
2017-06-09 17:50:01 +02:00
Ryan Ernst c1a3f50e19 Convert script uses to use source/id keys (elastic/x-pack-elasticsearch#1670)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25127

Original commit: elastic/x-pack-elasticsearch@e25bd90825
2017-06-09 08:29:36 -07:00
Lisa Cawley 15e4958d4e [DOCS] Update get datafeed statistics API output (elastic/x-pack-elasticsearch#1678)
Original commit: elastic/x-pack-elasticsearch@7e65102ac0
2017-06-09 07:48:51 -07:00
David Roberts 048ff24b79 [TEST] Wait for correct template version in rolling upgrade tests (elastic/x-pack-elasticsearch#1682)
We wait for index templates to be installed before running tests, but
these can get upgraded when the master node is upgraded.  If we don't
wait again in this case then tests can fail due to the cleanup code
being overwhelmed by pending cluster state updates that weren't waited
for before the test.

Original commit: elastic/x-pack-elasticsearch@eff0b24c11
2017-06-09 15:23:15 +01:00
David Kyle 02a241bdf5 [ML] Refresh index after delete (elastic/x-pack-elasticsearch#1675)
Original commit: elastic/x-pack-elasticsearch@fbbcd0343e
2017-06-09 12:27:30 +01:00
David Kyle d64bea14dc [ML] Closing an unknown job should throw resource not found exception (elastic/x-pack-elasticsearch#1673)
Original commit: elastic/x-pack-elasticsearch@c244d2809b
2017-06-09 10:02:51 +01:00
David Kyle d6e92c19da Fix closing/stopping unassigned jobs/datafeeds (elastic/x-pack-elasticsearch#1672)
Original commit: elastic/x-pack-elasticsearch@9f032ac98f
2017-06-09 09:33:36 +01:00
Lisa Cawley e29ab36849 [DOCS] Describe how to identify active ML model snapshot (elastic/x-pack-elasticsearch#1677)
Original commit: elastic/x-pack-elasticsearch@61766d0975
2017-06-08 16:02:41 -07:00
Chris Earle b31c8e2661 [Monitoring] Remove Specific ClusterStateCollector/Resolver and Tests (elastic/x-pack-elasticsearch#1664)
This removes the Cluster State collector and resolver and moves the collection of the cluster state (and cluster health, which is already included in cluster stats).

This makes the tests a little more stable and removes an extra network hop during monitoring data collection.

Original commit: elastic/x-pack-elasticsearch@44851d2dd6
2017-06-08 15:55:23 -04:00
Chris Earle 3f5ae2d54f Always use the lower version in resource tests
Original commit: elastic/x-pack-elasticsearch@47aa744acf
2017-06-08 15:39:52 -04:00
Nik Everett b8d86682ef Rework rolling restart tests (elastic/x-pack-elasticsearch#1590)
Reworks the rolling restart tests so that all configuration
options share code. Now there is a project per configuration:
* `qa:rolling-upgrade:with-ssl-with-system-key`
* `qa:rolling-upgrade:with-ssl-without-system-key`

Original commit: elastic/x-pack-elasticsearch@5672b4a4f2
2017-06-08 12:48:41 -06:00
jaymode dad15b8d6c Fix branch logic for bwc tests in the same major version
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.

Original commit: elastic/x-pack-elasticsearch@b07aa78a7b
2017-06-08 11:40:45 -06:00
David Roberts 64330ff14d Move bracket
(Matches part of a change made to 5.5/5.x that doesn't need fully forward
porting.)

Original commit: elastic/x-pack-elasticsearch@01f0837e6c
2017-06-08 18:01:58 +01:00
David Roberts 7cb1c8bd35 [TEST] Fix security test blacklist for new ML test
Original commit: elastic/x-pack-elasticsearch@b6a054a2a7
2017-06-08 10:12:33 +01:00
David Roberts f097ff906d Ensure only 1 in-flight request to create each watcher index template (elastic/x-pack-elasticsearch#1660)
The aim of this change is to prevent many identical requests to create
watcher index templates being submitted when a cluster first starts up
and many cluster state updates are happening.  Prior to this change, if
watcher's original index template creation requests queued up behind other
cluster state change requests then for each other request watcher would
re-request creation of all its index templates.  After this change it
uses a strategy similar to that used by ML to only have one creation
request per index template in the cluster state change queue at any time.

Relates elastic/x-pack-elasticsearch#1368
Relates elastic/x-pack-elasticsearch#1631
Relates elastic/x-pack-elasticsearch#1650

Original commit: elastic/x-pack-elasticsearch@ad87bf3f78
2017-06-08 10:05:39 +01:00
Tim Vernum 29c11c30f3 Settings filter should rely directly on realm settings (elastic/x-pack-elasticsearch#1669)
The security module had special cases for realm settings that should be filtered,
but since elastic/x-pack-elasticsearch@2c76a13 / elastic/elasticsearch#4311 it's possible to use the existing realm
setting objects to do that.

Original commit: elastic/x-pack-elasticsearch@0651afe987
2017-06-08 18:05:02 +10:00
David Roberts 7aa1114eca [ML] Make get_job_stats robust to missing results indices (elastic/x-pack-elasticsearch#1662)
Although the job stats for jobs with missing results indices are clearly
ruined, it's better to provide zeroes for the missing values and show the
stats for other jobs than to fail the whole request. This means the UI
can continue to function.

relates elastic/x-pack-elasticsearch#1656

Original commit: elastic/x-pack-elasticsearch@a06fa994a5
2017-06-08 08:33:06 +01:00
Jack Conradson d7658bd9a2 Generate Painless Factory for Creating Script Instances. (elastic/x-pack-elasticsearch#1667)
Original commit: elastic/x-pack-elasticsearch@bda1668eec
2017-06-07 16:06:30 -07:00
Chris Earle adc82e7323 make racey test a bit more flexible
Original commit: elastic/x-pack-elasticsearch@6d5c1110bf
2017-06-07 11:03:44 -04:00
David Kyle ba3e258470 [ML] Wait for job deletion if it is in the deleting state (elastic/x-pack-elasticsearch#1651)
* Wait for job deletion if it is in the deleting  state

* Tolerate errors if multiple force delete requests

Original commit: elastic/x-pack-elasticsearch@1f0c9fbb86
2017-06-07 15:41:29 +01:00
David Roberts f865755259 [ML] Tolerate InternalAssumptionViolatedException in overridden test (elastic/x-pack-elasticsearch#1657)
In order for elastic/elasticsearch#25100 to work, overridden test()
methods that call ESClientYamlSuiteTestCase.test() must not consume
any InternalAssumptionViolatedException that
ESClientYamlSuiteTestCase.test() throws.

Relates elastic/x-pack-elasticsearch#1650

Original commit: elastic/x-pack-elasticsearch@081ccaa0a6
2017-06-07 14:22:05 +01:00
Alexander Reelsen 887538d6bc Watcher: Fix BWC tests, disable test requiring upgrade API
Original commit: elastic/x-pack-elasticsearch@e8de71cdf7
2017-06-07 11:09:31 +02:00