Commit Graph

6703 Commits

Author SHA1 Message Date
Lisa Cawley a9b3cd747f [DOCS] Added ML forecast API (elastic/x-pack-elasticsearch#2745)
* [DOCS] Added ML forecast API

* [DOCS] Added forecast API to build.gradle

* [DOCS] Added forecast API example

* [DOCS] Fixed forecast API intro

* [DOCS] Addressed feedback on forecast API

* [DOCS] Added duration to forecast API

* [DOCS] Removed end time from forecast API

* [DOCS] Fixed gradle errors for forecast API

Original commit: elastic/x-pack-elasticsearch@db79e3d5bb
2017-11-23 11:52:37 -08:00
Alexander Reelsen 00ea3e8fc7 Watcher: Transform also needs to use stashing client (elastic/x-pack-elasticsearch#3098)
... in order to ensure that the xpack user is used.

Relates elastic/x-pack-elasticsearch#3054

Original commit: elastic/x-pack-elasticsearch@6edfcc3d87
2017-11-23 16:13:31 +01:00
Dimitris Athanasiou e0affd455d [ML] Change forecast_id to UUid, add create_time and start_time (elastic/x-pack-elasticsearch#3095)
relates elastic/x-pack-elasticsearch#3093

Original commit: elastic/x-pack-elasticsearch@f586189851
2017-11-23 14:46:52 +00:00
jaymode a8b5b138a7 Test: print cluster state xcontent on security index check failures
Original commit: elastic/x-pack-elasticsearch@0ff85f851c
2017-11-22 12:57:10 -07:00
lcawley 79cb3a8a2c [DOCS] Fixed typo
Original commit: elastic/x-pack-elasticsearch@a2da8f644f
2017-11-22 10:02:24 -08:00
Igor Kupczyński 18103fae7f Invalid value in the docs for transport.profiles...client_authentication (elastic/x-pack-elasticsearch#3091)
The 6.x and 6.0 versions of the documentation show

```yml
transport.profiles.client.xpack.security.ssl.client_authentication: no
```

Which results in 
```
2017-11-22T11:13:33,225][ERROR][org.elasticsearch.bootstrap.Bootstrap] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:452) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:392) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:142) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.node.Node.<init>(Node.java:302) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 14 more
Caused by: java.lang.IllegalArgumentException: could not resolve ssl client auth. unknown value [no]
	at org.elasticsearch.xpack.ssl.SSLClientAuth.parse(SSLClientAuth.java:78) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLConfigurationSettings.lambda$null$27(SSLConfigurationSettings.java:183) ~[?:?]
	at org.elasticsearch.common.settings.Setting.get(Setting.java:352) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.settings.Setting.get(Setting.java:346) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.xpack.ssl.SSLConfiguration.<init>(SSLConfiguration.java:80) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService.lambda$loadSSLConfigurations$1(SSLService.java:462) ~[?:?]
	at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_144]
	at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:461) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:87) ~[?:?]
	at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:237) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 14 more
```

I think the valid value there is `none`, so

```yml
transport.profiles.client.xpack.security.ssl.client_authentication: none
```

The tests seem to confirm that https://github.com/elastic/x-pack-elasticsearch/blob/elastic/x-pack-elasticsearch@4860e92d906e046a23aa07b39ee6ef637f011dc1/plugin/src/test/java/org/elasticsearch/xpack/ssl/SSLServiceTests.java#L269

Original commit: elastic/x-pack-elasticsearch@a35b3ac8c9
2017-11-22 17:09:06 +00:00
Jay Modi 0a683a0e18 Remove InternalClient and InternalSecurityClient (elastic/x-pack-elasticsearch#3054)
This change removes the InternalClient and the InternalSecurityClient. These are replaced with
usage of the ThreadContext and a transient value, `action.origin`, to indicate which component the
request came from. The security code has been updated to look for this value and ensure the
request is executed as the proper user. This work comes from elastic/x-pack-elasticsearch#2808 where @s1monw suggested
that we do this.

While working on this, I came across index template registries and rather than updating them to use
the new method, I replaced the ML one with the template upgrade framework so that we could
remove this template registry. The watcher template registry is still needed as the template must be
updated for rolling upgrades to work (see elastic/x-pack-elasticsearch#2950).

Original commit: elastic/x-pack-elasticsearch@7dbf2f263e
2017-11-22 08:35:18 -07:00
Alexander Reelsen c7a64667d4 Watcher: Properly url encode room names (elastic/x-pack-elasticsearch#2896)
Room names in hipchat were not properly URL encoded, thus room names
with spaces would not work as expected. This fixes all the hipchat
accounts by properly using spaces.

Also the hipchat tests are reenabled, as the IT team gave me new access to hipchat, 
allowing to create a fresh set of oauth tokens for the integration account type.

The HipchatServiceTests have also been converted to XPackSingleNodeTestCase

relates elastic/x-pack-elasticsearch#2371
relates elastic/x-pack-elasticsearch#2429

Original commit: elastic/x-pack-elasticsearch@9f8872f686
2017-11-22 15:50:18 +01:00
Dimitrios Athanasiou 02c83a3b6a [ML][TEST] Add delays between forecasts to ensure id uniqueness
relates elastic/x-pack-elasticsearch#3090

Original commit: elastic/x-pack-elasticsearch@5d6b091607
2017-11-22 12:04:44 +00:00
David Kyle 0dea758022 [ML] Log deprecation warning for jobs with delimited formats (elastic/x-pack-elasticsearch#3092)
Original commit: elastic/x-pack-elasticsearch@bd75fae990
2017-11-22 11:53:08 +00:00
Dimitris Athanasiou 74beb9ca64 [ML] Remove expired forecasts (elastic/x-pack-elasticsearch#3077)
Closes elastic/machine-learning-cpp#322


Original commit: elastic/x-pack-elasticsearch@5249452a86
2017-11-21 17:18:04 +00:00
Jay Modi 4ae1ca5fa5 Security: IndexLifecycleManager provides a consistent view of index state (elastic/x-pack-elasticsearch#3008)
This commit changes the IndexLifecycleManager's handling of variables about an index to only update
all of the values at a single time. Previously, all of the index state variables were volatile
members of the IndexLifecycleManager, which meant we could get an inconsistent view of the index
state. Although rare, this is still incorrect so this change adds a single volatile variable that
holds the state as of the last processed cluster state update.

Additionally, the IndexLifecycleManagerIntegTests were updated to have more concurrency and further
stress this portion of the code and its checks.

relates elastic/x-pack-elasticsearch#2973

Original commit: elastic/x-pack-elasticsearch@5f1552b298
2017-11-21 10:17:08 -07:00
Jay Modi d86e7870da Security: add manage_index_templates to the kibana_system role (elastic/x-pack-elasticsearch#3009)
This commit adds the manage_index_templates permission to the kibana_system role that is used by
the kibana system user. This is needed due to an upcoming feature in kibana where a index template
will be used to create the saved objects index.

relates elastic/x-pack-elasticsearch#2937

Original commit: elastic/x-pack-elasticsearch@85a67c73aa
2017-11-21 08:45:07 -07:00
David Roberts 35551859c5 [TEST] Fix side effects of elastic/x-pack-elasticsearch#2975 on build servers with very little RAM
Some of our REST tests open many jobs, and assuming each will use 1GB of
RAM on a single node cluster could fail the test.  The solution is to
explicitly say the test jobs will use very little RAM.

Original commit: elastic/x-pack-elasticsearch@a3fcfc4589
2017-11-21 15:11:28 +00:00
Hendrik Muhs cc66020cf3 [ML-FC] add expires_in parameter and change forecast_start_timestamp to timestamp (elastic/x-pack-elasticsearch#3073)
add expires_in parameter and change forecast_start_timestamp to timestamp

depends on elastic/machine-learning-cpp#421

Original commit: elastic/x-pack-elasticsearch@3a3eebd49c
2017-11-21 15:32:06 +01:00
Simon Willnauer 601222903d X-Pack side of elastic/elasticsearch#27469 (elastic/x-pack-elasticsearch#3071)
Original commit: elastic/x-pack-elasticsearch@99499b6bd6
2017-11-21 15:15:24 +01:00
Dimitris Athanasiou e71b5639de [ML] Rename id to forecast_id in forecast API response (elastic/x-pack-elasticsearch#3074)
Original commit: elastic/x-pack-elasticsearch@c05d9fc602
2017-11-21 13:57:41 +00:00
Dimitris Athanasiou 83ca6e8064 [ML] Report errors back to listener in DeleteExpiredDataAction (elastic/x-pack-elasticsearch#3072)
Currently, any errors that occur during the DeleteExpiredDataAction are logged and the deletion proceeds to the next job. The user will get no indication in the response that something went wrong although nothing should really go wrong unless the cluster is messed up.

This commit changes this so that errors are reported back to the action.

Original commit: elastic/x-pack-elasticsearch@489cf03c3e
2017-11-21 12:03:04 +00:00
Dimitris Athanasiou 754623753a [ML] Make it easier to add various ml data removal (elastic/x-pack-elasticsearch#3048)
Original commit: elastic/x-pack-elasticsearch@3e4ac6033b
2017-11-21 11:43:01 +00:00
David Roberts 402852a4ee Update BWC version after backporting to 6.1
Relates elastic/x-pack-elasticsearch#2975

Original commit: elastic/x-pack-elasticsearch@a63c56a019
2017-11-21 10:57:44 +00:00
Hendrik Muhs 41b254cdf4 change forecast message into an array of messages (elastic/x-pack-elasticsearch#3070)
depends on elastic/machine-learning-cpp#419

Turns the forecast message into an array of messages.

Original commit: elastic/x-pack-elasticsearch@7598342712
2017-11-21 11:47:34 +01:00
Luca Cavanna 941c0a5701 Cross Cluster Search: optionally skip disconnected clusters (elastic/x-pack-elasticsearch#2823)
Original commit: elastic/x-pack-elasticsearch@3b0017df1f
2017-11-21 11:42:39 +01:00
David Roberts f06acdc219 [ML] Improve the way ML jobs are allocated to nodes (elastic/x-pack-elasticsearch#2975)
This change modifies the way ML jobs are assigned to nodes to primarily
base the decision on the estimated memory footprint of the jobs. The
memory footprint comes from the model size stats if the job has been
running long enough, otherwise from the model memory limit. In addition,
an allowance for the program code and stack is added.

If insufficient information is available to base the allocation decision on
memory requirements then the decision falls back to using simple job
counts per node.

relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@b276aedf2f
2017-11-21 09:51:52 +00:00
Lisa Cawley 77803b06c5 [DOCS] Remove http.ssl.enabled step from installation (elastic/x-pack-elasticsearch#3065)
Original commit: elastic/x-pack-elasticsearch@b071fa191f
2017-11-20 11:29:13 -08:00
Lisa Cawley 712e740761 [DOCS] Added sysgenkey command and watcher encryption settings (elastic/x-pack-elasticsearch#3043)
* [DOCS] Added sysgenkey command and watcher settings

* [DOCS] Added data encryption task for Watcher

* [DOCS] Addressed feedback about watcher encryption

Original commit: elastic/x-pack-elasticsearch@edb1fccbfb
2017-11-20 08:44:43 -08:00
lcawley 5ec688a931 [DOCS] Removed beta warning from ML tutorial
Original commit: elastic/x-pack-elasticsearch@38d480fe06
2017-11-20 07:22:59 -08:00
David Kyle 040be7da9d [DOCS][TESTS] Don’t randomise content type for docs tests. (elastic/x-pack-elasticsearch#3058)
Original commit: elastic/x-pack-elasticsearch@7617901ff8
2017-11-20 13:32:14 +00:00
Michael Basnight f8cb1e603b Removed unused Terminal param (elastic/x-pack-elasticsearch#2854)
Relates elastic/elasticsearch#27216

Original commit: elastic/x-pack-elasticsearch@70bdda51ef
2017-11-19 22:33:27 -06:00
Michael Basnight 316da9a970 Move the CLI into its own subproject (elastic/x-pack-elasticsearch#3032)
relates elastic/elasticsearch#27114

Original commit: elastic/x-pack-elasticsearch@70e8488223
2017-11-18 21:43:25 -06:00
Nhat Nguyen 7c9af72d6e Removes the old handler for shard snapshot status
Relates https://github.com/elastic/elasticsearch/pull/27443

Original commit: elastic/x-pack-elasticsearch@7f0021e3f4
2017-11-17 20:14:44 -05:00
lcawley 32d0c1b0c7 [DOCS] Re-enabled code snippet testing
Original commit: elastic/x-pack-elasticsearch@31fd4c3668
2017-11-17 13:40:46 -08:00
lcawley a08ded367c Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@a66f322330
2017-11-17 13:30:20 -08:00
jaymode 34ecd18e76 Remove use of forbidden API URL#getPath
tool. However, this is a forbidden API so this commit replaces it with URI#getPath. Additionally,
the tests fail with a security manager permission error due to the use of Mockito for exception
throwing. This commit still uses Mockito for throwing exceptions but does it differently in a way
that is acceptable by our test security policy.

Original commit: elastic/x-pack-elasticsearch@5e1d45acf8
2017-11-17 14:03:26 -07:00
lcawley a7e0dfb3ec Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@209befd1b1
2017-11-17 12:58:43 -08:00
Albert Zaharovits 5b73c77011 SetupPasswordTool handle url cmd option correctly (elastic/x-pack-elasticsearch#2899)
Fixes bug when the url option had trailing slashes. The URL built
was invalid (consecutive fwd slashes) but the failure errors of
the subsequent requests were ignored.

URL is built correctly from the option spec.
True HTTP errors and Exceptions are logged and the cmd fails.

relates elastic/x-pack-elasticsearch#2778

Original commit: elastic/x-pack-elasticsearch@62b2d94ca0
2017-11-17 20:30:22 +02:00
Nhat Nguyen 418a6632e5 Registers a new handler for shard snapshot status
Relates elastic/x-pack-elasticsearch#27165

Original commit: elastic/x-pack-elasticsearch@11199642bb
2017-11-17 13:13:50 -05:00
lcawley 5dfb35e2a8 Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@f06f403c04
2017-11-17 10:03:54 -08:00
David Roberts 6741a62e80 [ML] Adjust memory limit test to account for greater accuracy (elastic/x-pack-elasticsearch#3047)
Due to elastic/machine-learning-cpp#409 the ML C++ code now instruments
memory more accurately.  This change modifies the expectations of the Java
integration test to account for the change.

Original commit: elastic/x-pack-elasticsearch@2ed7a75af4
2017-11-17 17:02:19 +00:00
Alexander Reelsen 0f97e28074 Watcher: Further preparations for source repo split (elastic/x-pack-elasticsearch#3006)
Changes to further prepare for feature split with regards to watcher:

- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
  used there
- The creation of the http client/notification classes have been moved
  into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
2017-11-17 17:05:07 +01:00
Alexander Reelsen 3d5fb54522 Watcher: Create templates on nodes newer than the master (elastic/x-pack-elasticsearch#2950)
This problem was introduced due to distributed watch execution.

When a node newer than the master node joins the cluster and gets a
.watches shard assigned it is supposed to start watcher. However
when a new version of the watch history template is part of that new
node (and we might increase that version anytime), this template does
not get installed, because only the master node is updating watcher
templates.

This commit checks if the local node version is higher than the master
node version and then also puts missing templates.

Currently this is done for all watcher templates, not only the watcher history.

relates elastic/x-pack-elasticsearch#2944

Original commit: elastic/x-pack-elasticsearch@4960231ea7
2017-11-17 16:56:29 +01:00
Alexander Reelsen 1933fc71f3 Tests: Mute more monitoring upgrade tests
Relates elastic/x-pack-elasticsearch#2948

Original commit: elastic/x-pack-elasticsearch@942ec95e6e
2017-11-17 13:54:50 +01:00
Hendrik Muhs a583cf270b [ML-FC] implement endpoint parameter that takes a duration (elastic/x-pack-elasticsearch#3027)
Adds a duration parameter to the forecast API. Also fixes issue if no parameter is given (forecast 1 day from time of last bucket), in which case it lets autodetect decide

depends on elastic/machine-learning-cpp#407

Original commit: elastic/x-pack-elasticsearch@3387478872
2017-11-17 06:37:03 +01:00
lcawley 7ae8f1cd9b [DOCS] Added sysgenkey command and watcher settings
Original commit: elastic/x-pack-elasticsearch@8e0b2027a6
2017-11-16 15:40:51 -08:00
lcawley 43a2572e7d [DOCS] Fixed typo in over_field_name
Original commit: elastic/x-pack-elasticsearch@c366d43448
2017-11-16 14:45:44 -08:00
Tim Brooks f94e6d427b Modify x-pack to for `TcpTransport` changes
This is related to elatic/elasticsearch#27407. That commit removed
parameterization from TcpTranport. This commit modifies the security
transport to be compatible with those changes.

Original commit: elastic/x-pack-elasticsearch@9878c26e14
2017-11-16 14:32:40 -07:00
Lisa Cawley 37c29a4006 [DOCS] Add categorization limitation (elastic/x-pack-elasticsearch#3022)
Original commit: elastic/x-pack-elasticsearch@387d7cf939
2017-11-16 10:05:11 -08:00
Hendrik Muhs 5efdb54eb8 add memoryUsage stats (elastic/x-pack-elasticsearch#3025)
adds memory usage statistics to forecast request stats

depends on: elastic/machine-learning-cpp#406

Original commit: elastic/x-pack-elasticsearch@abf90030fc
2017-11-16 18:38:13 +01:00
Dimitrios Athanasiou 80f4797f7a [ML][TEST] Add integ test for memory redution in population analysis
Original commit: elastic/x-pack-elasticsearch@ccd1f4835d
2017-11-16 17:10:59 +00:00
Zachary Tong 3c60c89504 [TEST] Skip "all" instead of "999"
The fake version was apparently making the yaml runner unhappy

Original commit: elastic/x-pack-elasticsearch@04193f9578
2017-11-16 01:29:00 +00:00
lcawley 9c0110678e [DOCS] Updated link to X-Pack breaking changes
Original commit: elastic/x-pack-elasticsearch@931bc10498
2017-11-15 16:39:05 -08:00