Commit Graph

6300 Commits

Author SHA1 Message Date
Tim Vernum a36121a725 [DOCS] [Security] Templates do not use bind_dn (elastic/x-pack-elasticsearch#1979)
Document that user_dn_template mode for LDAP authentication does not support bind_dn

Original commit: elastic/x-pack-elasticsearch@eef72615a8
2017-07-13 14:23:23 +10:00
Simon Willnauer 84ee21ed26 Followup for elastic/elasticsearch#25658 (elastic/x-pack-elasticsearch#1984)
This is the xpack side fo elastic/elasticsearch#25658 which is mainly refactorings
of a ctor and added tests.

Original commit: elastic/x-pack-elasticsearch@d8e2a2a057
2017-07-12 22:19:34 +02:00
Jay Modi e686d8a3bf Add active directory bind user and user lookup support (elastic/x-pack-elasticsearch#1956)
This commit adds support for a bind user when using the active directory realm. The addition of a
bind user also enables support for the user lookup mechanism, which is necessary to support the run
as functionality that we provide.

relates elastic/x-pack-elasticsearch#179

Original commit: elastic/x-pack-elasticsearch@40b07b3422
2017-07-12 14:01:39 -06:00
Brandon Kobel ef25568b2a Removing indices privileges from the reporting_user (elastic/x-pack-elasticsearch#1806)
* Removing indices privileges from the reporting_user

* Updating tests

* Adding comment

Original commit: elastic/x-pack-elasticsearch@ba7688b1dd
2017-07-12 15:56:35 -04:00
Alexander Reelsen e64cf23b13 Watcher: Start watcher on master node only with mixed versions (elastic/x-pack-elasticsearch#1983)
When there are data or master nodes in the cluster, that are older
than ES 6.0 alpha3, then watcher will only start on the master node.

Changed all transport actions to be master node actions, as there is
already a method to decide to run locally, which we can piggyback on.

Original commit: elastic/x-pack-elasticsearch@65cecb6d69
2017-07-12 20:58:47 +02:00
Tim Sullivan 51adbf0f51 [Monitoring] Update template version to 6.0.0 beta1 version (elastic/x-pack-elasticsearch#1948)
* [Monitoring] Update template version to 6000003

* update LAST_UPDATED_VERSION

* use the 6.0.0-beta1 version constant

* use the 6.0.0-beta1 version value (6000026)

* fix cluster uuid reference in terms filters

Original commit: elastic/x-pack-elasticsearch@40c822d396
2017-07-12 09:19:54 -07:00
David Roberts 176786de54 [ML] Improve mapping clash error message (elastic/x-pack-elasticsearch#1968)
relates elastic/x-pack-elasticsearch#1751

Original commit: elastic/x-pack-elasticsearch@01c221bf42
2017-07-12 15:58:37 +01:00
Jack Conradson 32dbfba0c2 Disallow lang to specified in requests where a stored script is used. (elastic/x-pack-elasticsearch#1949)
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.

Original commit: elastic/x-pack-elasticsearch@926a7b2d86
2017-07-12 07:56:08 -07:00
Igor Motov 41f30b0ae4 Upgrade API: add common upgrade check method (elastic/x-pack-elasticsearch#1970)
Moves internal index format check to a central location.

Original commit: elastic/x-pack-elasticsearch@0869baac51
2017-07-12 08:46:59 -04:00
Hendrik Muhs f5cfb98ea2 do not update mapping if it is newer than the current version (elastic/x-pack-elasticsearch#1971)
In case of mixed version in a cluster the update could potentially downgrade the mappings. This CL changes the check to avoid this situation.

Original commit: elastic/x-pack-elasticsearch@9ecd5df70e
2017-07-12 09:19:06 +02:00
Chris Earle 753690bf5c [TEST][Monitoring] Use specific LAST_UPDATED constant to avoid issues when they're not the same (elastic/x-pack-elasticsearch#1976)
Original commit: elastic/x-pack-elasticsearch@a41bff564a
2017-07-11 20:47:36 -04:00
Chris Earle 44eda0962e [Monitoring] Use ctx.metadata.xpack not just ctx.metadata (elastic/x-pack-elasticsearch#1974)
This tweaks a recent fix so that it uses the right metadata variable.

Original commit: elastic/x-pack-elasticsearch@af9d9b0e41
2017-07-11 18:04:38 -04:00
Jason Tedor 5ed683538d Name the log tail thread and mark it as a daemon
This commit provides an explicit name for the log tail thread, otherwise
the thread shows up with a generic name like Thread-12 in stack
dumps. While the stack trace provides enough information to conclude
what this thread is doing, it is better to be more explicit about the
purpose of this thread so that that can be discerned directly from the
name of the thread.

Additionally, we mark this thread as a daemon thread. Since this thread
is created by the main thread during node initialization it defaults to
being a user thread. Since the JVM only exits when the only threads
running are not user threads, if this thread were to somehow block then
it could prevent the server JVM from exiting. As such, this thread
should be marked as a daemon thread.

Relates elastic/x-pack-elasticsearch#1969

Original commit: elastic/x-pack-elasticsearch@b3761c7d91
2017-07-11 16:05:07 -04:00
Tim Brooks c22f618a1a Modify password tool to pull password from keystore (elastic/x-pack-elasticsearch#1951)
This is related to elastic/x-pack-elasticsearch#1217. This commit modifies the password setup tool
to pull the password for the `elastic` user from the keystore.

Original commit: elastic/x-pack-elasticsearch@fbc71f393d
2017-07-11 12:50:53 -05:00
Igor Motov 9030c3ae77 Upgrade API: remove kibana upgraders (elastic/x-pack-elasticsearch#1963)
Also removes handling of extra parameters in the upgrade api

Original commit: elastic/x-pack-elasticsearch@9edb825da7
2017-07-11 11:29:31 -04:00
Jay Modi 03ed2bbbd0 Add setting for the LDAP user search filter and deprecate user attribute (elastic/x-pack-elasticsearch#1959)
This commit adds a setting to allow changing the user search filter. Previously the filter was a
simple equality filter that mapped a given attribute to the value of the username. The default
behavior remains the same with this change but provides additional flexibility to users to who may
need more advanced LDAP searches. The user attribute setting has been deprecated due to the overlap
with the new filter setting.

relates elastic/x-pack-elasticsearch#1861

Original commit: elastic/x-pack-elasticsearch@e9d797e81c
2017-07-11 09:27:24 -06:00
Alexander Reelsen 22b0560ccb Watcher: Ensure no template deletion race in mixed cluster environment (elastic/x-pack-elasticsearch#1964)
When having a mixed cluster with 5.6 and 6.0 nodes, the template upgrade
service has a cluster state listener that deletes the old watches and
triggered_watches index template. However during that time the 5.6 node
WatcherIndexTemplateRegistry checks if the templates are missing and
adds them back. This results in a race, because the new .watches index
template does not get added by the WatcherIndexTemplateRegistry when the
6.0 node is node a master node.

This commit circumvents this issue, by only deleting the watches and
triggered watches template during the upgrade process.

Original commit: elastic/x-pack-elasticsearch@71380f460a
2017-07-11 16:50:41 +02:00
David Roberts 5288791f41 [ML] Specify only concrete indices during deletion (elastic/x-pack-elasticsearch#1966)
Following elastic/elasticsearch#23997 this was only working due to
the way we were suppressing certain errors during job deletion.

This PR makes the situations we want to ignore during job deletion
clearer and adheres to the intention of elastic/elasticsearch#23997
by only specifying concrete indices to the `indices` arguments of
deletion calls.

Original commit: elastic/x-pack-elasticsearch@2458c3db40
2017-07-11 14:07:48 +01:00
Adrien Grand 2e7c725d03 Fix compile errors in src/test.
Original commit: elastic/x-pack-elasticsearch@2ab67bbbdc
2017-07-11 14:31:35 +02:00
Alexander Reelsen 82499b6e55 Fixing compilation issue after lucene snapshot update
Original commit: elastic/x-pack-elasticsearch@1d9e73ad9a
2017-07-11 14:27:21 +02:00
Clinton Gormley 81101b893a Added note to cross cluster search docs to specify minimum node version of 5.5
Original commit: elastic/x-pack-elasticsearch@98e440f1a4
2017-07-11 14:15:23 +02:00
Yannick Welsch 6b02197ca1 [TEST] Ensure stable cluster before disabling licensing in testEnableDisableBehaviour
If the cluster is still making cluster state updates while disabling the license, cluster state updates might not go through,
triggering an assertion failure at the end of the test that checks if all cluster states have been applied.

relates elastic/x-pack-elasticsearch#1627

Original commit: elastic/x-pack-elasticsearch@e11863fd02
2017-07-11 11:48:27 +02:00
Simon Willnauer 16f2de017d Trivial refactorings after elastic/elasticsearch#25627
Original commit: elastic/x-pack-elasticsearch@1cb9c2f959
2017-07-11 11:42:51 +02:00
Hendrik Muhs 261d65a9c4 [ML] add meta data field into mappings and update procedure (elastic/x-pack-elasticsearch#1925)
Adds a update procedure for ML index mappings in order to allow adding new fields. The ES version is stored in the "_meta" field of the ML mappings which then get applied to every index. When opening a job, this version is checked on the (shared) index and a mapping update is performed in case that the version is older than current.

Original commit: elastic/x-pack-elasticsearch@211608c7ad
2017-07-11 07:44:41 +02:00
Alexander Reelsen 14c12cfcde Revert "Tests: Disable all watcher BWC tests until fixed correctly"
This reverts commit elastic/x-pack-elasticsearch@8043ec1858.

Original commit: elastic/x-pack-elasticsearch@39180b095c
2017-07-11 00:40:05 +02:00
Alexander Reelsen 4dc13c3698 Tests: Disable all watcher BWC tests until fixed correctly
Original commit: elastic/x-pack-elasticsearch@8043ec1858
2017-07-10 23:12:19 +02:00
Tal Levy 4fdfa5cf6f add index checks (elastic/x-pack-elasticsearch#1938)
* add index checks

The following checks apply to both 5.x as well as 6.0

index checks:
- coercion of boolean fields
- the _all meta field is now disabled by default
- the include_in_all mapping parameter is now disallowed
- unrecognized match_mapping_type options not silently ignored
- similarity settings
- store settings
- store throttling settings
- shadow replicas have been removed

Original commit: elastic/x-pack-elasticsearch@5583b21763
2017-07-10 12:19:02 -07:00
Tim Brooks f9eabcdf08 Load bootstrap elastic user password from keystore (elastic/x-pack-elasticsearch#1942)
This is related to elastic/x-pack-elasticsearch#1217. This commit adds a ClusterStateListener at
node startup. Once the cluster and security index are ready, this
listener will attempt to set the elastic user's password with the
bootstrap password pulled from the keystore. If the password is not in
the keystore or the elastic password has already been set, nothing will
be done.

Original commit: elastic/x-pack-elasticsearch@7fc4943c45
2017-07-10 11:15:39 -05:00
Lisa Cawley 31b02c3941 [DOCS] Update model_memory_limit (elastic/x-pack-elasticsearch#1928)
* [DOCS] Update model_memory_limit

* [DOCS] Clarify minimum model_memory_limit value

* [DOCS] More updates to model_memory_limit

* [DOCS] Address feedback in jobresource.asciidoc

Original commit: elastic/x-pack-elasticsearch@3c62719037
2017-07-10 08:50:38 -07:00
Lisa Cawley 7674729bbe [DOCS] Add ML tribe node limitation (elastic/x-pack-elasticsearch#1947)
Original commit: elastic/x-pack-elasticsearch@c447fcd899
2017-07-10 08:08:09 -07:00
Alexander Reelsen 1a57120a6b Security: Add index template permissions for system user (elastic/x-pack-elasticsearch#1937)
As the TemplateUpgradeService requires permissions to add and
delete index template, we have to grant those to the _system user.

This commit adds such permissions plus an integration test.

Original commit: elastic/x-pack-elasticsearch@a76ca9c738
2017-07-10 16:15:13 +02:00
Colin Goodheart-Smithe f70f432695 hanges to accompany agg package structure change in ES
Original commit: elastic/x-pack-elasticsearch@5b79242656
2017-07-10 11:57:11 +01:00
Alexander Reelsen bc5be696e4 Monitoring: Fix cluster monitoring watches to prevent compilations (elastic/x-pack-elasticsearch#1944)
The monitoring watches are roughly executing the same queries even when
they run against different clusters. However the way they were created,
where the cluster name is replaced via search & replace instead of using
watch metadata implies, that every watch is different from a script
compilation cache perspective. On top of that every of those watches is
executed once a minute. So if a new node becomes master and you monitor
three clusters, this results in a fair share of compilations in the first minute. The
reason for the compilation is the fact, that the search input uses
mustache for being able to add dynamic parts into a search using
mustache.

Several of those watches also need to compile more than one search
request.

The maximum default value for script compilations is only 15 and thus at
least one watch will not be executed due to failing script compilations.

This commit changes the four watches, so that the search requests are
cacheable. This means, no matter how many clusters you monitor, there
will be only needed four compilations for the different watches and
that's it.

Relates elastic/support-dev-help#2090

Original commit: elastic/x-pack-elasticsearch@6c877421bb
2017-07-10 09:20:16 +02:00
Jason Tedor c0305727c2 Bump version to 6.0.0-beta1
This commit follows core where the version was bumped from 6.0.0-alpha3
to 6.0.0-beta1 and the 6.0.0-alpha3 version constant was replaced by
6.0.0-beta1.

Relates elastic/x-pack-elasticsearch#1950

Original commit: elastic/x-pack-elasticsearch@a7816fec70
2017-07-09 18:12:31 -04:00
Igor Motov 37075bd201 Upgrade API: Add end-to-end bwc test for watcher upgrade (elastic/x-pack-elasticsearch#1939)
This test creates watches in old versions of elasticsearch, upgrades them after upgrading cluster to the latest version and then tests that they were upgraded correctly.

Original commit: elastic/x-pack-elasticsearch@b9d45eb2a5
2017-07-09 11:21:57 -04:00
Boaz Leskes 04125b5ca2 Adapt create_bwc_indexes.py to new monitoring index structure
Original commit: elastic/x-pack-elasticsearch@a881f78eb2
2017-07-08 09:03:34 +02:00
Nik Everett 65a0d44b2f Handle "current" being dropped from the list of released versions
Original commit: elastic/x-pack-elasticsearch@46668ce672
2017-07-07 16:01:21 -04:00
Nik Everett 1af9f56834 Remove `IndexResponse.created`
Core removed this in elastic/elasticsearch#25516.

Original commit: elastic/x-pack-elasticsearch@b1e7040fa2
2017-07-07 13:57:43 -04:00
Lisa Cawley 2e58164b96 [DOCS] Add ML limitation (elastic/x-pack-elasticsearch#1916)
Original commit: elastic/x-pack-elasticsearch@7730810f98
2017-07-07 10:17:55 -07:00
Lisa Cawley 2aa53f253f [DOCS] Update doc links for graph API (elastic/x-pack-elasticsearch#1884)
Original commit: elastic/x-pack-elasticsearch@9eebb6c9a6
2017-07-07 10:11:06 -07:00
Tim Sullivan 66ebdff447 [Monitoring] Email actions for Cluster Alerts (elastic/x-pack-elasticsearch#1879)
* [Monitoring] Email actions for Cluster Alerts

* fix quotations in email fields

* move email vars to transform, and rename for snake_case

* add state to email subject for cluster status alert

* remove types field in kibana_settings search

* simplify email action condition script

* uppercase the state for the email subject

* only append state to email subject if alert is new

* show state in email subject even when alert is resolved

Original commit: elastic/x-pack-elasticsearch@e6fdd8d620
2017-07-07 09:55:45 -07:00
Simon Willnauer c87d9278a6 Add validation for all `transport.profile.*` settings (elastic/x-pack-elasticsearch#1909)
Follow-up from elasticsearch/elastic#25508

Original commit: elastic/x-pack-elasticsearch@fe08e74ccc
2017-07-07 09:41:34 +02:00
Tim Vernum c5012ac6e8 [DOC] Miscellaneous security doc updates (elastic/x-pack-elasticsearch#1908)
- Document refresh interval for role mapping files
- Fix obsolete shield reference in transport profile example 
- Clarify that AD & PKI don't support run_as
- Fix logstash conf examples
- Clarify interaction of SSL settings and PKI realm settings
- Document PKI DN format, and recommend use of pki_dn metadata
- Provide more details about action.auto_create_index during setup

Original commit: elastic/x-pack-elasticsearch@49ddb12a7e
2017-07-07 13:33:35 +10:00
Jason Tedor b636dcc366 Upgrade to Netty 4.1.13.Final
This commit updates the SHAs here as core upgraded the Netty dependency
from 4.1.11.Final to 4.1.13.Final.

Relates elastic/x-pack-elasticsearch#1936

Original commit: elastic/x-pack-elasticsearch@d1803fc331
2017-07-06 15:37:17 -04:00
Lisa Cawley 0e322b525d [DOCS] Update doc links for security APIs (elastic/x-pack-elasticsearch#1883)
Original commit: elastic/x-pack-elasticsearch@660410008c
2017-07-06 11:06:03 -07:00
Tim Brooks d95c365e64 Loosen setup mode restrictions for upgrade tests (elastic/x-pack-elasticsearch#1927)
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.

This commit loosens two aspects of the security.

1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.

Original commit: elastic/x-pack-elasticsearch@8a2a577038
2017-07-06 10:37:48 -05:00
Dimitris Athanasiou 4e03c657a3 [ML] Write model_memory_limit with units into the cluster state (elastic/x-pack-elasticsearch#1934)
This is step 2 of elastic/x-pack-elasticsearch#1604

This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.

Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
2017-07-06 15:40:19 +01:00
David Roberts 984d2ca2ba [ML] Ignore unknown fields when parsing ML cluster state (elastic/x-pack-elasticsearch#1924)
ML has two types of custom cluster state:

1. jobs
2. datafeeds

These need to be parsed from JSON in two situations:

1. Create/update of the job/datafeed
2. Restoring cluster state on startup

Previously we used exactly the same parser in both situations, but this
severely limits our ability to add new features.  This is because the
parser was very strict.  This was good when accepting create/update
requests from users, but when restoring cluster state from disk it meant
that we could not add new fields, as that would prevent reloading in
mixed version clusters.

This commit introduces a second parser that tolerates unknown fields for
each object that is stored in cluster state.  Then we use this more
tolerant parser when parsing cluster state, but still use the strict
parser when parsing REST requests.

relates elastic/x-pack-elasticsearch#1732

Original commit: elastic/x-pack-elasticsearch@754e51d1ec
2017-07-06 13:36:11 +01:00
Dimitris Athanasiou adc6fd5a0f [ML] Parse model_memory_limit both as number and string with units (elastic/x-pack-elasticsearch#1921)
This is the first step for elastic/x-pack-elasticsearch#1604.

Original commit: elastic/x-pack-elasticsearch@70010a216d
2017-07-06 12:29:12 +01:00
Lee Hinman b66560fa85 [TEST] Use better REST endpoints for testing watch actions
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/24437

It changes two things, for the disable tests, it uses a valid endpoint instead
of a previously invalid endpoint that happened to return a 400 because the
endpoint was bad, regardless of if watcher was disabled.

The other change is to create the watches index by putting a watch using the
correct API, rather than manually creating the index. This is because
`RestHijackOperationAction` hijacks operations like this and stops accessing the
endpoint in a regular manner.

Original commit: elastic/x-pack-elasticsearch@3be78d9aea
2017-07-05 10:55:25 -06:00