Commit Graph

6692 Commits

Author SHA1 Message Date
Jay Modi 4ae1ca5fa5 Security: IndexLifecycleManager provides a consistent view of index state (elastic/x-pack-elasticsearch#3008)
This commit changes the IndexLifecycleManager's handling of variables about an index to only update
all of the values at a single time. Previously, all of the index state variables were volatile
members of the IndexLifecycleManager, which meant we could get an inconsistent view of the index
state. Although rare, this is still incorrect so this change adds a single volatile variable that
holds the state as of the last processed cluster state update.

Additionally, the IndexLifecycleManagerIntegTests were updated to have more concurrency and further
stress this portion of the code and its checks.

relates elastic/x-pack-elasticsearch#2973

Original commit: elastic/x-pack-elasticsearch@5f1552b298
2017-11-21 10:17:08 -07:00
Jay Modi d86e7870da Security: add manage_index_templates to the kibana_system role (elastic/x-pack-elasticsearch#3009)
This commit adds the manage_index_templates permission to the kibana_system role that is used by
the kibana system user. This is needed due to an upcoming feature in kibana where a index template
will be used to create the saved objects index.

relates elastic/x-pack-elasticsearch#2937

Original commit: elastic/x-pack-elasticsearch@85a67c73aa
2017-11-21 08:45:07 -07:00
David Roberts 35551859c5 [TEST] Fix side effects of elastic/x-pack-elasticsearch#2975 on build servers with very little RAM
Some of our REST tests open many jobs, and assuming each will use 1GB of
RAM on a single node cluster could fail the test.  The solution is to
explicitly say the test jobs will use very little RAM.

Original commit: elastic/x-pack-elasticsearch@a3fcfc4589
2017-11-21 15:11:28 +00:00
Hendrik Muhs cc66020cf3 [ML-FC] add expires_in parameter and change forecast_start_timestamp to timestamp (elastic/x-pack-elasticsearch#3073)
add expires_in parameter and change forecast_start_timestamp to timestamp

depends on elastic/machine-learning-cpp#421

Original commit: elastic/x-pack-elasticsearch@3a3eebd49c
2017-11-21 15:32:06 +01:00
Simon Willnauer 601222903d X-Pack side of elastic/elasticsearch#27469 (elastic/x-pack-elasticsearch#3071)
Original commit: elastic/x-pack-elasticsearch@99499b6bd6
2017-11-21 15:15:24 +01:00
Dimitris Athanasiou e71b5639de [ML] Rename id to forecast_id in forecast API response (elastic/x-pack-elasticsearch#3074)
Original commit: elastic/x-pack-elasticsearch@c05d9fc602
2017-11-21 13:57:41 +00:00
Dimitris Athanasiou 83ca6e8064 [ML] Report errors back to listener in DeleteExpiredDataAction (elastic/x-pack-elasticsearch#3072)
Currently, any errors that occur during the DeleteExpiredDataAction are logged and the deletion proceeds to the next job. The user will get no indication in the response that something went wrong although nothing should really go wrong unless the cluster is messed up.

This commit changes this so that errors are reported back to the action.

Original commit: elastic/x-pack-elasticsearch@489cf03c3e
2017-11-21 12:03:04 +00:00
Dimitris Athanasiou 754623753a [ML] Make it easier to add various ml data removal (elastic/x-pack-elasticsearch#3048)
Original commit: elastic/x-pack-elasticsearch@3e4ac6033b
2017-11-21 11:43:01 +00:00
David Roberts 402852a4ee Update BWC version after backporting to 6.1
Relates elastic/x-pack-elasticsearch#2975

Original commit: elastic/x-pack-elasticsearch@a63c56a019
2017-11-21 10:57:44 +00:00
Hendrik Muhs 41b254cdf4 change forecast message into an array of messages (elastic/x-pack-elasticsearch#3070)
depends on elastic/machine-learning-cpp#419

Turns the forecast message into an array of messages.

Original commit: elastic/x-pack-elasticsearch@7598342712
2017-11-21 11:47:34 +01:00
Luca Cavanna 941c0a5701 Cross Cluster Search: optionally skip disconnected clusters (elastic/x-pack-elasticsearch#2823)
Original commit: elastic/x-pack-elasticsearch@3b0017df1f
2017-11-21 11:42:39 +01:00
David Roberts f06acdc219 [ML] Improve the way ML jobs are allocated to nodes (elastic/x-pack-elasticsearch#2975)
This change modifies the way ML jobs are assigned to nodes to primarily
base the decision on the estimated memory footprint of the jobs. The
memory footprint comes from the model size stats if the job has been
running long enough, otherwise from the model memory limit. In addition,
an allowance for the program code and stack is added.

If insufficient information is available to base the allocation decision on
memory requirements then the decision falls back to using simple job
counts per node.

relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@b276aedf2f
2017-11-21 09:51:52 +00:00
Lisa Cawley 77803b06c5 [DOCS] Remove http.ssl.enabled step from installation (elastic/x-pack-elasticsearch#3065)
Original commit: elastic/x-pack-elasticsearch@b071fa191f
2017-11-20 11:29:13 -08:00
Lisa Cawley 712e740761 [DOCS] Added sysgenkey command and watcher encryption settings (elastic/x-pack-elasticsearch#3043)
* [DOCS] Added sysgenkey command and watcher settings

* [DOCS] Added data encryption task for Watcher

* [DOCS] Addressed feedback about watcher encryption

Original commit: elastic/x-pack-elasticsearch@edb1fccbfb
2017-11-20 08:44:43 -08:00
lcawley 5ec688a931 [DOCS] Removed beta warning from ML tutorial
Original commit: elastic/x-pack-elasticsearch@38d480fe06
2017-11-20 07:22:59 -08:00
David Kyle 040be7da9d [DOCS][TESTS] Don’t randomise content type for docs tests. (elastic/x-pack-elasticsearch#3058)
Original commit: elastic/x-pack-elasticsearch@7617901ff8
2017-11-20 13:32:14 +00:00
Michael Basnight f8cb1e603b Removed unused Terminal param (elastic/x-pack-elasticsearch#2854)
Relates elastic/elasticsearch#27216

Original commit: elastic/x-pack-elasticsearch@70bdda51ef
2017-11-19 22:33:27 -06:00
Michael Basnight 316da9a970 Move the CLI into its own subproject (elastic/x-pack-elasticsearch#3032)
relates elastic/elasticsearch#27114

Original commit: elastic/x-pack-elasticsearch@70e8488223
2017-11-18 21:43:25 -06:00
Nhat Nguyen 7c9af72d6e Removes the old handler for shard snapshot status
Relates https://github.com/elastic/elasticsearch/pull/27443

Original commit: elastic/x-pack-elasticsearch@7f0021e3f4
2017-11-17 20:14:44 -05:00
lcawley 32d0c1b0c7 [DOCS] Re-enabled code snippet testing
Original commit: elastic/x-pack-elasticsearch@31fd4c3668
2017-11-17 13:40:46 -08:00
lcawley a08ded367c Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@a66f322330
2017-11-17 13:30:20 -08:00
jaymode 34ecd18e76 Remove use of forbidden API URL#getPath
tool. However, this is a forbidden API so this commit replaces it with URI#getPath. Additionally,
the tests fail with a security manager permission error due to the use of Mockito for exception
throwing. This commit still uses Mockito for throwing exceptions but does it differently in a way
that is acceptable by our test security policy.

Original commit: elastic/x-pack-elasticsearch@5e1d45acf8
2017-11-17 14:03:26 -07:00
lcawley a7e0dfb3ec Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@209befd1b1
2017-11-17 12:58:43 -08:00
Albert Zaharovits 5b73c77011 SetupPasswordTool handle url cmd option correctly (elastic/x-pack-elasticsearch#2899)
Fixes bug when the url option had trailing slashes. The URL built
was invalid (consecutive fwd slashes) but the failure errors of
the subsequent requests were ignored.

URL is built correctly from the option spec.
True HTTP errors and Exceptions are logged and the cmd fails.

relates elastic/x-pack-elasticsearch#2778

Original commit: elastic/x-pack-elasticsearch@62b2d94ca0
2017-11-17 20:30:22 +02:00
Nhat Nguyen 418a6632e5 Registers a new handler for shard snapshot status
Relates elastic/x-pack-elasticsearch#27165

Original commit: elastic/x-pack-elasticsearch@11199642bb
2017-11-17 13:13:50 -05:00
lcawley 5dfb35e2a8 Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@f06f403c04
2017-11-17 10:03:54 -08:00
David Roberts 6741a62e80 [ML] Adjust memory limit test to account for greater accuracy (elastic/x-pack-elasticsearch#3047)
Due to elastic/machine-learning-cpp#409 the ML C++ code now instruments
memory more accurately.  This change modifies the expectations of the Java
integration test to account for the change.

Original commit: elastic/x-pack-elasticsearch@2ed7a75af4
2017-11-17 17:02:19 +00:00
Alexander Reelsen 0f97e28074 Watcher: Further preparations for source repo split (elastic/x-pack-elasticsearch#3006)
Changes to further prepare for feature split with regards to watcher:

- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
  used there
- The creation of the http client/notification classes have been moved
  into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
2017-11-17 17:05:07 +01:00
Alexander Reelsen 3d5fb54522 Watcher: Create templates on nodes newer than the master (elastic/x-pack-elasticsearch#2950)
This problem was introduced due to distributed watch execution.

When a node newer than the master node joins the cluster and gets a
.watches shard assigned it is supposed to start watcher. However
when a new version of the watch history template is part of that new
node (and we might increase that version anytime), this template does
not get installed, because only the master node is updating watcher
templates.

This commit checks if the local node version is higher than the master
node version and then also puts missing templates.

Currently this is done for all watcher templates, not only the watcher history.

relates elastic/x-pack-elasticsearch#2944

Original commit: elastic/x-pack-elasticsearch@4960231ea7
2017-11-17 16:56:29 +01:00
Alexander Reelsen 1933fc71f3 Tests: Mute more monitoring upgrade tests
Relates elastic/x-pack-elasticsearch#2948

Original commit: elastic/x-pack-elasticsearch@942ec95e6e
2017-11-17 13:54:50 +01:00
Hendrik Muhs a583cf270b [ML-FC] implement endpoint parameter that takes a duration (elastic/x-pack-elasticsearch#3027)
Adds a duration parameter to the forecast API. Also fixes issue if no parameter is given (forecast 1 day from time of last bucket), in which case it lets autodetect decide

depends on elastic/machine-learning-cpp#407

Original commit: elastic/x-pack-elasticsearch@3387478872
2017-11-17 06:37:03 +01:00
lcawley 7ae8f1cd9b [DOCS] Added sysgenkey command and watcher settings
Original commit: elastic/x-pack-elasticsearch@8e0b2027a6
2017-11-16 15:40:51 -08:00
lcawley 43a2572e7d [DOCS] Fixed typo in over_field_name
Original commit: elastic/x-pack-elasticsearch@c366d43448
2017-11-16 14:45:44 -08:00
Tim Brooks f94e6d427b Modify x-pack to for `TcpTransport` changes
This is related to elatic/elasticsearch#27407. That commit removed
parameterization from TcpTranport. This commit modifies the security
transport to be compatible with those changes.

Original commit: elastic/x-pack-elasticsearch@9878c26e14
2017-11-16 14:32:40 -07:00
Lisa Cawley 37c29a4006 [DOCS] Add categorization limitation (elastic/x-pack-elasticsearch#3022)
Original commit: elastic/x-pack-elasticsearch@387d7cf939
2017-11-16 10:05:11 -08:00
Hendrik Muhs 5efdb54eb8 add memoryUsage stats (elastic/x-pack-elasticsearch#3025)
adds memory usage statistics to forecast request stats

depends on: elastic/machine-learning-cpp#406

Original commit: elastic/x-pack-elasticsearch@abf90030fc
2017-11-16 18:38:13 +01:00
Dimitrios Athanasiou 80f4797f7a [ML][TEST] Add integ test for memory redution in population analysis
Original commit: elastic/x-pack-elasticsearch@ccd1f4835d
2017-11-16 17:10:59 +00:00
Zachary Tong 3c60c89504 [TEST] Skip "all" instead of "999"
The fake version was apparently making the yaml runner unhappy

Original commit: elastic/x-pack-elasticsearch@04193f9578
2017-11-16 01:29:00 +00:00
lcawley 9c0110678e [DOCS] Updated link to X-Pack breaking changes
Original commit: elastic/x-pack-elasticsearch@931bc10498
2017-11-15 16:39:05 -08:00
lcawley ff54c4190a [DOCS] Fixed links to X-Pack Kibana release notes
Original commit: elastic/x-pack-elasticsearch@57159e6bc6
2017-11-15 15:28:44 -08:00
lcawley e4518871ee Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@6a2df70985
2017-11-15 15:27:04 -08:00
Zachary Tong 34898f2717 [TEST] AwaitsFix/Skip REST test "qa/rolling_upgrade/60_monitoring"
Test appears to be broken and fails consistently.  See elastic/x-pack-elasticsearch#2948

Original commit: elastic/x-pack-elasticsearch@e3ba8277e3
2017-11-15 20:35:44 +00:00
Tim Brooks d45c2d784c Update x-pack to support TcpChannel (elastic/x-pack-elasticsearch#2983)
This is related to elastic/elasticsearch#27132. This commit updates
x-pack to support signature changes introduced in that PR.

Original commit: elastic/x-pack-elasticsearch@461c96ffe6
2017-11-15 12:38:56 -07:00
lcawley 10a9334046 Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@7af3facb7d
2017-11-15 09:09:42 -08:00
Lisa Cawley f8497dc943 [DOCS] Added flow diagram for installation (elastic/x-pack-elasticsearch#3020)
Original commit: elastic/x-pack-elasticsearch@c7c39d9724
2017-11-15 09:08:48 -08:00
Lee Hinman 889b008298 Add "client-api-objects" dependency for xpack plugin and transport-client (elastic/x-pack-elasticsearch#2995)
* Add "client-api-objects" dependency for xpack plugin and transport-client

This adds another gradle project, "client-api-objects" which is intended to be a
common dependency so that the xpack plugin and transport-client can share the
same Request and Response objects.

Relates to elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a6d83299d0
2017-11-15 09:49:00 -07:00
Tim Brooks 5d47b96905 Modify x-pack to for `TransportChannel` changes (elastic/x-pack-elasticsearch#3013)
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#27388. It modifies x-pack to
be compatible with the removal of the delegating transport channel.

Original commit: elastic/x-pack-elasticsearch@3bd7bf6773
2017-11-15 09:48:17 -07:00
lcawley 0eee69b6d3 [DOCS] Added flow diagram for installation
Original commit: elastic/x-pack-elasticsearch@048ac11a17
2017-11-15 08:27:12 -08:00
Alexander Reelsen 36105231c3 Watcher: Return useful error message when no accounts are found (elastic/x-pack-elasticsearch#2897)
When there were no accounts configured, watcher returned a cryptic
error message containing 'null' in the description. This fix returns
a more clear error message.

On top a dedicated NotificationServiceTests class was added to remove
redundant test cases in the hipchat/jira/slack unit tests, that all
basically tested NotificationService capabilties.

relates elastic/x-pack-elasticsearch#2666

Original commit: elastic/x-pack-elasticsearch@45d0d1df31
2017-11-15 11:51:23 +01:00
Alexander Reelsen f286f9b0f2 Watcher: Remove unused test helper class
Original commit: elastic/x-pack-elasticsearch@09f66a5f52
2017-11-15 11:05:59 +01:00