Commit Graph

6088 Commits

Author SHA1 Message Date
Tim Vernum c63d4e306b Reduce logging for reserved realm authc failures (elastic/x-pack-elasticsearch#1711)
We recently added logging for critical authentication failures as they had previously been silent (with respect to logs) but would cause authentication processing to stop.

However, the reserved realm intentionally uses exceptions to stop any other realm authenticating a reserved user if the password is entered incorrectly.
Since this is the most common use of exceptions in the authc chain, we reduce the logging verbosity in normal cases (drop the stack trace, remove "unexpected") and only log the full details in debug.

Original commit: elastic/x-pack-elasticsearch@686a98010b
2017-06-16 10:46:07 +10:00
Martijn van Groningen 8cc4f29f33 test: make sure analysis-common module is also available on the cluster 1 and cluster2 nodes.
Original commit: elastic/x-pack-elasticsearch@eef5d2b566
2017-06-15 23:25:45 +02:00
jaymode ee3d17adfd Test: fix security rolling upgrade tests that were failing
This commit fixes the default password migration tests that had been failing reproducibly. The
first fix skips tests using the set enabled api when running against a version prior to 5.1.2 as
this api would otherwise trip an assertion that the xcontent builder was not closed. The second
fix is to ensure we set the password field in the user object.

relates elastic/x-pack-elasticsearch#1529
relates elastic/x-pack-elasticsearch#1516

Original commit: elastic/x-pack-elasticsearch@2f9c804309
2017-06-15 14:50:26 -06:00
Martijn van Groningen 863755d2da Make sure that the module are installed too.
Original commit: elastic/x-pack-elasticsearch@f581d0902c
2017-06-15 22:03:18 +02:00
Martijn van Groningen 9dddce2e08 Add analysis-common as test dependency to xpack plugin module.
Original commit: elastic/x-pack-elasticsearch@2472585037
2017-06-15 20:50:31 +02:00
Tim Brooks 7c7e47aa0f Add tool to setup passwords for internal users (elastic/x-pack-elasticsearch#1434)
This is related to elastic/x-pack-elasticsearch#1217. This change introduces a tool
bin/x-pack/setup-passwords that will streamline the setting of
internal user passwords. There are two modes of operation. One mode
called auto, automatically generates passwords and prints them to
the console. The second mode called interactive allows the user to 
enter passwords.

All passwords are changed using the elastic superuser. The elastic
password is the first password to be set.

Original commit: elastic/x-pack-elasticsearch@00974234a2
2017-06-15 10:48:02 -05:00
Jay Modi d920cc7348 Remove signing code completely from master (elastic/x-pack-elasticsearch#1719)
After improving the authorization of scroll requests and backporting to 5.x, we no longer need to
have any signing code in master. This commit removes it.

Original commit: elastic/x-pack-elasticsearch@8b65fd9338
2017-06-15 07:51:39 -06:00
Jim Ferenczi 07fcf75dd9 [Test] Fix DocumentLevelSecurityTests and FieldLevelSecurityTests to work with the new format of the parent-join field
Original commit: elastic/x-pack-elasticsearch@cd9f26887b
2017-06-15 15:40:41 +02:00
David Roberts 291aa27562 [ML] Kill autodetect on error writing headers (elastic/x-pack-elasticsearch#1730)
If an exception occurs while sending the initial setup messages to the autodetect
such that it fails rather than reaching the open state then the autodetect process
needs to be killed to prevent it hogging resources.

Relates elastic/x-pack-elasticsearch#1684

Original commit: elastic/x-pack-elasticsearch@1ee80ed9b0
2017-06-15 14:03:41 +01:00
David Roberts b748da1880 [ML] Prevent time_field and control field name in analysis_config (elastic/x-pack-elasticsearch#1729)
In does not make sense for the time_field in the data_description to
be used as a by/over/partition field name, nor the summary_count_field,
categorization_field or as an influencer.  Therefore, configurations
where the time_field in the data_description is used in the
analysis_config are now rejected.

Additionally, it causes a problem communicating with the C++ code if
the control field name (which is '.') is used in the analysis_config,
so this is also rejected at the validation stage.

Relates elastic/x-pack-elasticsearch#1684

Original commit: elastic/x-pack-elasticsearch@e6750a2cda
2017-06-15 13:04:25 +01:00
Jason Tedor 8c5e7b589c Use master flag for disabling BWC tests
This commit skips the the BWC tests if the master BWC flag
bwc_tests_enabled in core is set to false.

Relates elastic/x-pack-elasticsearch#1725

Original commit: elastic/x-pack-elasticsearch@7b924066a9
2017-06-15 07:45:20 -04:00
Boaz Leskes c76bdf61d7 move assertBusy to use CheckException (elastic/x-pack-elasticsearch#1727)
We use assertBusy in many places where the underlying code throw exceptions. Currently we need to wrap those exceptions in a RuntimeException which is ugly.

Companion PR to https://github.com/elastic/elasticsearch/pull/25246

Original commit: elastic/x-pack-elasticsearch@056857273f
2017-06-15 13:24:34 +02:00
Tim Vernum e36f86cf95 Improve upgrade process for reserved users (elastic/x-pack-elasticsearch#1712)
- Don't attempt to upgrade from 2.x
- Attempt up to 10 retries if the migration fails (with increasing back-off between attempts)
- If a cached user is disabled, recheck with the underlying store

The last change is required if the migration takes a long time.
While users are being migrated, they might be marked as disabled, but when the migration is complete they need to be usable immediately.

Original commit: elastic/x-pack-elasticsearch@2621867014
2017-06-15 10:27:23 +10:00
Lisa Cawley 429f7ca7a3 [DOCS] Build ES Reference from x-pack-elasticsearch (elastic/x-pack-elasticsearch#1722)
* [DOCS] Build ES Reference from x-pack-elasticsearch

* [DOCS] Added es-test-dir attribute

* [DOCS] Added plugins-examples-dir attribute

Original commit: elastic/x-pack-elasticsearch@05714eb8ca
2017-06-14 17:01:45 -07:00
Nik Everett 0970c509bc Remove the assemble task from projects not published (elastic/x-pack-elasticsearch#1721)
Removes the `assemble` task from projects that aren't published
to speed up `gradle assemble` so the unified release can call it.

Original commit: elastic/x-pack-elasticsearch@43dfcc15f3
2017-06-14 19:57:26 -04:00
Igor Motov 8abf4c3119 Tests: Remove QUERY_AND_FETCH BWC actions from the list of known handlers
Relates to elastic/elasticsearch#25223

Original commit: elastic/x-pack-elasticsearch@2da097d1ea
2017-06-14 19:01:14 -04:00
jaymode d299592c9d Docs: clean up incorrect path in certgen output in the SSL docs
Original commit: elastic/x-pack-elasticsearch@0d2102094c
2017-06-14 14:31:17 -06:00
Simon Willnauer 97693b9357 Add scroll support for cross cluster search (elastic/x-pack-elasticsearch#1706)
Original commit: elastic/x-pack-elasticsearch@eadffa396b
2017-06-14 20:38:58 +02:00
Jay Modi 9c8e12280b Test: increase the wait for green cluster health calls (elastic/x-pack-elasticsearch#1703)
This commit increases the amount of time to wait for green cluster health during a rolling upgrade
to account for the time that may be needed in the case of delayed shards. Additionally some old
timeout values were removed as they were used due to the default timeout of 30s.

Relates elastic/x-pack-elasticsearch#1683

Original commit: elastic/x-pack-elasticsearch@9996673db0
2017-06-14 10:25:40 -06:00
Lisa Cawley 1157d3adf9 [DOCS] Prepare XPack APIs for Elasticsearch Ref (elastic/x-pack-elasticsearch#1715)
Original commit: elastic/x-pack-elasticsearch@5e75581952
2017-06-14 09:21:58 -07:00
Dimitrios Athanasiou 50de99374c [TEST] Start non ML node in BasicDistributedJobsIT.testMlIndicesNotAvailable
Original commit: elastic/x-pack-elasticsearch@71e63b6ea2
2017-06-14 16:43:57 +01:00
Dimitrios Athanasiou 1eb785d8ee [TEST] Minor refactorings in BasicDistributedJobsIT
Original commit: elastic/x-pack-elasticsearch@eab0fe759e
2017-06-14 16:04:06 +01:00
Dimitrios Athanasiou cd119c488f [TEST] Remove test logging for resolved issue elastic/x-pack-elasticsearch#1048
Original commit: elastic/x-pack-elasticsearch@012d2b53c9
2017-06-14 15:21:20 +01:00
David Kyle 1010f73ae7 [ML] Retry after broken scroll (elastic/x-pack-elasticsearch#1713)
Original commit: elastic/x-pack-elasticsearch@b4fc329c52
2017-06-14 15:04:14 +01:00
Alexander Reelsen 4e085f03b7 Tests: Disable watcher in ServerTransportFilterIntegrationTests
These tests are starting their own nodes and do not use the testing
trigger schedule class.

There are occasional test failure due to a race condition where watcher
is in the process of being started, but cannot be shut down properly,
because starting up was not finished when the shut down was called for.

These filter tests do not rely on watcher, so we can disable them for
now, but we still need to fix a race condition in starting/stopping
watcher.

relates elastic/x-pack-elasticsearch#1422

Original commit: elastic/x-pack-elasticsearch@f13bb7a6fb
2017-06-14 15:00:37 +02:00
Chris Earle bb31c09ed0 [Monitoring] Add Mapping for cfs_quota_micros field (elastic/x-pack-elasticsearch#1710)
This maps the cgroup field, which was unmapped (and now mappings are no longer dynamic).

Original commit: elastic/x-pack-elasticsearch@86b4333ebb
2017-06-14 07:37:15 -04:00
Alexander Reelsen 457af97d1f Watcher: Remove handling of _status field (elastic/x-pack-elasticsearch#1701)
There is no need to handle any _status field in
the 6.0 release from now on, as everything has been
taken care in the upgrade API.

Original commit: elastic/x-pack-elasticsearch@606581f4d7
2017-06-14 10:36:46 +02:00
David Roberts e7af52fb8e [ML] Let core ES stop the native controller when ML is disabled (elastic/x-pack-elasticsearch#1704)
This changes part of the logic that was added in elastic/x-pack-elasticsearch#644 and extended
in elastic/x-pack-elasticsearch#1495 so that when ML is disabled we never try to communicate with
the native controller during node shutdown.

The original reason for needing to communicate with the native controller
when ML is disabled was the problem of elastic/prelert-legacy#803.
However, this was fixed in a better way in elastic/elasticsearch#24579.
Now there is considerable benefit in never talking to the native
controller from the plugin code when ML is disabled, because it means
anyone suffering some obscure problem with ML can disable it without
running the risk of uncovering some other obscure problem with shutdown.

Original commit: elastic/x-pack-elasticsearch@9d329483a7
2017-06-14 08:54:57 +01:00
Alexander Reelsen 4346431156 Tests: Ensure watcher is started in REST tests (elastic/x-pack-elasticsearch#1702)
This adds a check in the REST tests to ensure that
watcher is started, and if not, tries to start watcher.

This eliminates test failures where watcher was not in
the correct state due to other tests stopping watcher.

Original commit: elastic/x-pack-elasticsearch@fc547d49b4
2017-06-14 08:34:09 +02:00
Ryan Ernst abe217ebc3 Remove uses of Strings.cleanPath (elastic/x-pack-elasticsearch#1709)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25209

Original commit: elastic/x-pack-elasticsearch@4f59f011e9
2017-06-13 21:10:02 -07:00
Lisa Cawley 9311d056ad [DOCS] Further doc build clarifications in README
Original commit: elastic/x-pack-elasticsearch@47393500b7
2017-06-13 14:55:45 -07:00
Lisa Cawley 4ab141d0d0 [DOCS] Add documentation build info to README
Original commit: elastic/x-pack-elasticsearch@b42ae5d8d4
2017-06-13 14:36:35 -07:00
Lisa Cawley dcc4fd28da [DOCS] Change standalone node to dedicated node (elastic/x-pack-elasticsearch#1708)
Original commit: elastic/x-pack-elasticsearch@41e13a2de1
2017-06-13 14:04:15 -07:00
Lisa Cawley f84ae92e51 [DOCS] Add X-Pack node settings for Elasticsearch Reference (elastic/x-pack-elasticsearch#1688)
* [DOCS] Add X-Pack node settings for Elasticsearch Reference

* [DOCS] Fix callout in node.asciidoc

Original commit: elastic/x-pack-elasticsearch@9c29441734
2017-06-13 12:13:29 -07:00
Suyog Rao fe72991c70 [Logstash] Add new component to x-pack to handle LS features (elastic/x-pack-elasticsearch#1530)
This commit adds a new Logstash component to x-pack to support the config management work. Currently, the functionality in this component is really simple; all it does is upload a new index template for `.logstash` index. This index stores the actual LS configuration.

On this template is bootstrapped in ES, Kibana can write user-created LS configs which adhere to the mapping defined here. In the future, we're looking into adding more functionality on the ES side to handle config documents, but for now, this is simple.

relates elastic/x-pack-elasticsearch#1499, relates elastic/x-pack-elasticsearch#1471

Original commit: elastic/x-pack-elasticsearch@d7cc8675f7
2017-06-13 10:30:30 -07:00
Dimitris Athanasiou f2e2ccae01 [ML] Extract parent field when job has text multi-field (elastic/x-pack-elasticsearch#1705)
In the case where a field is a text multi-field, it has
no doc values and it is not in source. Thus, the datafeed
will not be able to extract it.

However, it is possible to extract it by getting its parent
field instead. This commit implements the logic to look
in parent fields when the field in question is a text field.

Original commit: elastic/x-pack-elasticsearch@f116e89921
2017-06-13 18:00:24 +01:00
Clinton Gormley fff33e753a Corrected syntax for adding IDs to definition lists in built-in roles
Relates to elastic/x-pack-elasticsearch#5040

Original commit: elastic/x-pack-elasticsearch@693c4adfb5
2017-06-13 14:00:56 +02:00
Clinton Gormley 5a0ffb4b3a Fixed bad asciidoc link
relates elastic/x-pack-elasticsearch#5040

Original commit: elastic/x-pack-elasticsearch@feb5afbee4
2017-06-13 11:54:06 +02:00
Simon Willnauer 65bac10eed Fix API change in SecurityNetty4Transport
Original commit: elastic/x-pack-elasticsearch@f152fb1813
2017-06-13 10:15:26 +02:00
Alexander Reelsen b86c2e6e18 Watcher: Fix ack watch documentation (elastic/x-pack-elasticsearch#1695)
The ack watch documentation was exlcuded from the tests and
thus contained wrong information. This commit cleans it up.

relates elastic/x-pack-elasticsearch#1642

Original commit: elastic/x-pack-elasticsearch@81d69174df
2017-06-13 09:57:10 +02:00
Lisa Cawley 62ee1bc635 [DOCS] Add ML categorization of messages (elastic/x-pack-elasticsearch#1666)
* [DOCS] Add ML categorization of messages

* [DOCS] Describe ML categorization_examples_limit property

* [DOCS] Updated ML categorization of messages

* [DOCS] Add links to ML categorization

Original commit: elastic/x-pack-elasticsearch@6403f6ce84
2017-06-12 10:41:14 -07:00
Dimitrios Athanasiou 29811ea1d8 [ML] Write null instead of false for optional boolean that was removed
Original commit: elastic/x-pack-elasticsearch@43f5fc04bd
2017-06-12 17:35:25 +01:00
Dimitrios Athanasiou 1d33fb8b8b [TEST] Add now required mapping for data index in datafeed test
Original commit: elastic/x-pack-elasticsearch@67cf68c5a0
2017-06-12 15:50:16 +01:00
James Baiera 1d3921f581 Fixing vagrant build file issues with new vagrant support plugin
Original commit: elastic/x-pack-elasticsearch@dbb5d4a215
2017-06-12 10:03:30 -04:00
Dimitris Athanasiou 8eb62eac27 [ML] Automate detection of way to extract fields (elastic/x-pack-elasticsearch#1686)
In 5.4.x, the datafeed attempts to get all fields from
doc_values by default. It has a `_source` parameter which
when enabled changes the strategy to instead try to get
all fields from the source.

This has been the most common issue users have been
reporting as it means the datafeed will fail to fetch
any text fields by default.

This change uses the field capabilities API in order
to automatically detect whether a field is aggregatable.
It then extracts such fields from doc_values while the
rest are taken from source. The change also adds
validation to the start datafeed action so that if
fields are missing mappings or the time field is not
aggregatable we respond with an appropriate error.

relates elastic/x-pack-elasticsearch#1649

Original commit: elastic/x-pack-elasticsearch@76e2cc6cb2
2017-06-12 14:56:31 +01:00
Dimitris Athanasiou c9bbc17742 [ML] Make datafeed types optional (elastic/x-pack-elasticsearch#1690)
Specifying types for a datafeed should be optional
as no types is equal to searching through all types.

Original commit: elastic/x-pack-elasticsearch@f61ac01b45
2017-06-12 14:46:53 +01:00
Alexander Reelsen 27b5142de6 Watcher: Fix croneval tool for packaging (elastic/x-pack-elasticsearch#1689)
The croneval script used an old parameter to start up.
This commit removes this parameter, that is used, when a
package is used.

In addition a concrete vagrant test has been added.

relates elastic/x-pack-elasticsearch#1635

Original commit: elastic/x-pack-elasticsearch@ea7b8a08f4
2017-06-12 13:56:26 +02:00
Alexander Reelsen 83a5e022cb Watcher: Reduce logging noise when watcher might be stopped (elastic/x-pack-elasticsearch#1685)
Only log an entry when an actual stop is executed instead of
always logging.

Also added a reason to stop watcher to the methods, so that
debug logs will yield that information.

Original commit: elastic/x-pack-elasticsearch@8efaed0e9a
2017-06-12 10:24:16 +02:00
David Kyle 516696bbec [ML][TEST] Refresh after indexing docs
Original commit: elastic/x-pack-elasticsearch@b64ad1b2c6
2017-06-10 19:12:52 +01:00
Boaz Leskes b5ab68fac8 qa/upgrade_cluster/10_basic.yaml add shard level info on health failure
Original commit: elastic/x-pack-elasticsearch@46847ca262
2017-06-09 21:59:46 +02:00