Commit Graph

6716 Commits

Author SHA1 Message Date
Lisa Cawley 5507c46257 [DOCS] Fixed cross cluster search docs issue (elastic/x-pack-elasticsearch#3113)
Original commit: elastic/x-pack-elasticsearch@023d220082
2017-11-27 07:56:38 -08:00
Alexander Reelsen 6406c9816a Watcher: Add transform input for chained input (elastic/x-pack-elasticsearch#2861)
The chained input in watcher is a useful feature to
call several endpoints before execution a condition.
However it was pretty hard to modify data from a previous
input in order to be able to execute it in another input.

This commit adds a another input, called a `transform` input,
which allows you to do a transform as another input in a chained
input.

See this example

```
"input" : {
  "chain" : {
    "inputs" : [ <1>
      {
        "first" : {
          "simple" : { "path" : "/_search" }
        }
      },
      {
        "second" : {
          "transform" : {
            "script" : "return [ 'path' : 'ctx.payload.first.path' + '/' ]"
          }
        }
      },
      {
        "third" : {
          "http" : {
            "request" : {
              "host" : "localhost",
              "port" : 9200,
              "path" : "{{ctx.payload.second.path}}" <2>
            }
          }
        }
      }
    ]
  }
}
```

This allows for far more flexibility before executing the next input in a chained
one.

Original commit: elastic/x-pack-elasticsearch@3af9ba6e9b
2017-11-27 13:27:56 +01:00
David Kyle 3957518ab2 [DOCS] Fix cleaning up ML resources after tests
Original commit: elastic/x-pack-elasticsearch@986bdcd1f3
2017-11-27 10:02:16 +00:00
David Roberts 304330e1bc [ML] Specify ML_ORIGIN when calling the job update endpoint internally (elastic/x-pack-elasticsearch#3110)
This change applies the same pattern that was applied in elastic/x-pack-elasticsearch#3054 to a new
method that was introduced in elastic/x-pack-elasticsearch#2975 which was in-flight at the same time
so missed the original batch of changes.

relates elastic/x-pack-elasticsearch#3109

Original commit: elastic/x-pack-elasticsearch@8076c6cf6a
2017-11-27 09:18:24 +00:00
Jason Tedor c777c1d36f Refactor CLI commands as logging-aware commands
We have to ensure logging is configured for any CLI command that depends
on core Elasticsearch (since it might directly or indirectly touch
logging). This commit does this for all commands in X-Pack.

Relates elastic/x-pack-elasticsearch#3112

Original commit: elastic/x-pack-elasticsearch@f77f9b5052
2017-11-25 11:40:29 -05:00
Dimitris Athanasiou eb4186dd5c [ML] Stop datafeed when job fails (elastic/x-pack-elasticsearch#3107)
The problem here was that when the autodetect process crashes
we set the job state to FAILED but we did not remove the
communicator from the map in AutodetectProcessManager.

relates elastic/x-pack-elasticsearch#2773

Original commit: elastic/x-pack-elasticsearch@9b8eafb4d0
2017-11-24 15:04:29 +00:00
Alexander Reelsen d89d8abec9 Watcher: Fix equals/hashcode for WatchStatus (elastic/x-pack-elasticsearch#3105)
This was missed in elastic/x-pack-elasticsearch#3103 and fixes to add the headers variable to both
methods to ensure comparisons work as expected.

Original commit: elastic/x-pack-elasticsearch@df5e422698
2017-11-24 15:32:25 +01:00
Russ Cam e4e8870b13 Add opening state to Job states (elastic/x-pack-elasticsearch#2317)
Also updated open state to opened.

Original commit: elastic/x-pack-elasticsearch@663d95db1a
2017-11-24 11:35:51 +00:00
Alexander Reelsen 4fe9ac734b Watcher: Store thread context headers in watch (elastic/x-pack-elasticsearch#2808)
In order to be able to execute a watch as the user, who stored the
watch, this commit stores certain headers of the thread context, that
was used when the watch was stored.

Upon loading the watch the headers are loaded and applied for the
following watcher execution features

* search transform
* search input
* index action

A special case is the execute watch API, which overrides the headers loaded
from the watch with the one of the current request, so that a user
cannot execute this watch with other privileges of the user who stored it.

Only the headers "es-security-runas-user", "_xpack_security_authentication" are
copied for now, as those are needed for our security features.

The headers are stored in watch status in the watch and are not returned by default,
when the GET Watch API is used. A search reveals those of course.

relates elastic/x-pack-elasticsearch#2201

Original commit: elastic/x-pack-elasticsearch@9803bd51c2
2017-11-24 09:15:54 +01:00
David Turner 933c22dce6 Tidy up after elastic/x-pack-elasticsearch#3078 (elastic/x-pack-elasticsearch#3099)
Original commit: elastic/x-pack-elasticsearch@ebe599a422
2017-11-24 08:13:13 +00:00
lcawley 4d24748170 [DOCS] Added requirement to forecast API
Original commit: elastic/x-pack-elasticsearch@3f1360ca2b
2017-11-23 15:54:04 -08:00
lcawley 1f808ad580 [DOCS] Fixed broken link to forecast API
Original commit: elastic/x-pack-elasticsearch@41d9918459
2017-11-23 13:05:26 -08:00
David Turner 3e8b3491d5 Consolidate version numbering semantics (elastic/x-pack-elasticsearch#3078)
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.

Original commit: elastic/x-pack-elasticsearch@a1d456f30a
2017-11-23 20:23:05 +00:00
Lisa Cawley a9b3cd747f [DOCS] Added ML forecast API (elastic/x-pack-elasticsearch#2745)
* [DOCS] Added ML forecast API

* [DOCS] Added forecast API to build.gradle

* [DOCS] Added forecast API example

* [DOCS] Fixed forecast API intro

* [DOCS] Addressed feedback on forecast API

* [DOCS] Added duration to forecast API

* [DOCS] Removed end time from forecast API

* [DOCS] Fixed gradle errors for forecast API

Original commit: elastic/x-pack-elasticsearch@db79e3d5bb
2017-11-23 11:52:37 -08:00
Alexander Reelsen 00ea3e8fc7 Watcher: Transform also needs to use stashing client (elastic/x-pack-elasticsearch#3098)
... in order to ensure that the xpack user is used.

Relates elastic/x-pack-elasticsearch#3054

Original commit: elastic/x-pack-elasticsearch@6edfcc3d87
2017-11-23 16:13:31 +01:00
Dimitris Athanasiou e0affd455d [ML] Change forecast_id to UUid, add create_time and start_time (elastic/x-pack-elasticsearch#3095)
relates elastic/x-pack-elasticsearch#3093

Original commit: elastic/x-pack-elasticsearch@f586189851
2017-11-23 14:46:52 +00:00
jaymode a8b5b138a7 Test: print cluster state xcontent on security index check failures
Original commit: elastic/x-pack-elasticsearch@0ff85f851c
2017-11-22 12:57:10 -07:00
lcawley 79cb3a8a2c [DOCS] Fixed typo
Original commit: elastic/x-pack-elasticsearch@a2da8f644f
2017-11-22 10:02:24 -08:00
Igor Kupczyński 18103fae7f Invalid value in the docs for transport.profiles...client_authentication (elastic/x-pack-elasticsearch#3091)
The 6.x and 6.0 versions of the documentation show

```yml
transport.profiles.client.xpack.security.ssl.client_authentication: no
```

Which results in 
```
2017-11-22T11:13:33,225][ERROR][org.elasticsearch.bootstrap.Bootstrap] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:452) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:392) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:142) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.node.Node.<init>(Node.java:302) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 14 more
Caused by: java.lang.IllegalArgumentException: could not resolve ssl client auth. unknown value [no]
	at org.elasticsearch.xpack.ssl.SSLClientAuth.parse(SSLClientAuth.java:78) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLConfigurationSettings.lambda$null$27(SSLConfigurationSettings.java:183) ~[?:?]
	at org.elasticsearch.common.settings.Setting.get(Setting.java:352) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.settings.Setting.get(Setting.java:346) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.xpack.ssl.SSLConfiguration.<init>(SSLConfiguration.java:80) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService.lambda$loadSSLConfigurations$1(SSLService.java:462) ~[?:?]
	at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_144]
	at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:461) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:87) ~[?:?]
	at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:237) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 14 more
```

I think the valid value there is `none`, so

```yml
transport.profiles.client.xpack.security.ssl.client_authentication: none
```

The tests seem to confirm that https://github.com/elastic/x-pack-elasticsearch/blob/elastic/x-pack-elasticsearch@4860e92d906e046a23aa07b39ee6ef637f011dc1/plugin/src/test/java/org/elasticsearch/xpack/ssl/SSLServiceTests.java#L269

Original commit: elastic/x-pack-elasticsearch@a35b3ac8c9
2017-11-22 17:09:06 +00:00
Jay Modi 0a683a0e18 Remove InternalClient and InternalSecurityClient (elastic/x-pack-elasticsearch#3054)
This change removes the InternalClient and the InternalSecurityClient. These are replaced with
usage of the ThreadContext and a transient value, `action.origin`, to indicate which component the
request came from. The security code has been updated to look for this value and ensure the
request is executed as the proper user. This work comes from elastic/x-pack-elasticsearch#2808 where @s1monw suggested
that we do this.

While working on this, I came across index template registries and rather than updating them to use
the new method, I replaced the ML one with the template upgrade framework so that we could
remove this template registry. The watcher template registry is still needed as the template must be
updated for rolling upgrades to work (see elastic/x-pack-elasticsearch#2950).

Original commit: elastic/x-pack-elasticsearch@7dbf2f263e
2017-11-22 08:35:18 -07:00
Alexander Reelsen c7a64667d4 Watcher: Properly url encode room names (elastic/x-pack-elasticsearch#2896)
Room names in hipchat were not properly URL encoded, thus room names
with spaces would not work as expected. This fixes all the hipchat
accounts by properly using spaces.

Also the hipchat tests are reenabled, as the IT team gave me new access to hipchat, 
allowing to create a fresh set of oauth tokens for the integration account type.

The HipchatServiceTests have also been converted to XPackSingleNodeTestCase

relates elastic/x-pack-elasticsearch#2371
relates elastic/x-pack-elasticsearch#2429

Original commit: elastic/x-pack-elasticsearch@9f8872f686
2017-11-22 15:50:18 +01:00
Dimitrios Athanasiou 02c83a3b6a [ML][TEST] Add delays between forecasts to ensure id uniqueness
relates elastic/x-pack-elasticsearch#3090

Original commit: elastic/x-pack-elasticsearch@5d6b091607
2017-11-22 12:04:44 +00:00
David Kyle 0dea758022 [ML] Log deprecation warning for jobs with delimited formats (elastic/x-pack-elasticsearch#3092)
Original commit: elastic/x-pack-elasticsearch@bd75fae990
2017-11-22 11:53:08 +00:00
Dimitris Athanasiou 74beb9ca64 [ML] Remove expired forecasts (elastic/x-pack-elasticsearch#3077)
Closes elastic/machine-learning-cpp#322


Original commit: elastic/x-pack-elasticsearch@5249452a86
2017-11-21 17:18:04 +00:00
Jay Modi 4ae1ca5fa5 Security: IndexLifecycleManager provides a consistent view of index state (elastic/x-pack-elasticsearch#3008)
This commit changes the IndexLifecycleManager's handling of variables about an index to only update
all of the values at a single time. Previously, all of the index state variables were volatile
members of the IndexLifecycleManager, which meant we could get an inconsistent view of the index
state. Although rare, this is still incorrect so this change adds a single volatile variable that
holds the state as of the last processed cluster state update.

Additionally, the IndexLifecycleManagerIntegTests were updated to have more concurrency and further
stress this portion of the code and its checks.

relates elastic/x-pack-elasticsearch#2973

Original commit: elastic/x-pack-elasticsearch@5f1552b298
2017-11-21 10:17:08 -07:00
Jay Modi d86e7870da Security: add manage_index_templates to the kibana_system role (elastic/x-pack-elasticsearch#3009)
This commit adds the manage_index_templates permission to the kibana_system role that is used by
the kibana system user. This is needed due to an upcoming feature in kibana where a index template
will be used to create the saved objects index.

relates elastic/x-pack-elasticsearch#2937

Original commit: elastic/x-pack-elasticsearch@85a67c73aa
2017-11-21 08:45:07 -07:00
David Roberts 35551859c5 [TEST] Fix side effects of elastic/x-pack-elasticsearch#2975 on build servers with very little RAM
Some of our REST tests open many jobs, and assuming each will use 1GB of
RAM on a single node cluster could fail the test.  The solution is to
explicitly say the test jobs will use very little RAM.

Original commit: elastic/x-pack-elasticsearch@a3fcfc4589
2017-11-21 15:11:28 +00:00
Hendrik Muhs cc66020cf3 [ML-FC] add expires_in parameter and change forecast_start_timestamp to timestamp (elastic/x-pack-elasticsearch#3073)
add expires_in parameter and change forecast_start_timestamp to timestamp

depends on elastic/machine-learning-cpp#421

Original commit: elastic/x-pack-elasticsearch@3a3eebd49c
2017-11-21 15:32:06 +01:00
Simon Willnauer 601222903d X-Pack side of elastic/elasticsearch#27469 (elastic/x-pack-elasticsearch#3071)
Original commit: elastic/x-pack-elasticsearch@99499b6bd6
2017-11-21 15:15:24 +01:00
Dimitris Athanasiou e71b5639de [ML] Rename id to forecast_id in forecast API response (elastic/x-pack-elasticsearch#3074)
Original commit: elastic/x-pack-elasticsearch@c05d9fc602
2017-11-21 13:57:41 +00:00
Dimitris Athanasiou 83ca6e8064 [ML] Report errors back to listener in DeleteExpiredDataAction (elastic/x-pack-elasticsearch#3072)
Currently, any errors that occur during the DeleteExpiredDataAction are logged and the deletion proceeds to the next job. The user will get no indication in the response that something went wrong although nothing should really go wrong unless the cluster is messed up.

This commit changes this so that errors are reported back to the action.

Original commit: elastic/x-pack-elasticsearch@489cf03c3e
2017-11-21 12:03:04 +00:00
Dimitris Athanasiou 754623753a [ML] Make it easier to add various ml data removal (elastic/x-pack-elasticsearch#3048)
Original commit: elastic/x-pack-elasticsearch@3e4ac6033b
2017-11-21 11:43:01 +00:00
David Roberts 402852a4ee Update BWC version after backporting to 6.1
Relates elastic/x-pack-elasticsearch#2975

Original commit: elastic/x-pack-elasticsearch@a63c56a019
2017-11-21 10:57:44 +00:00
Hendrik Muhs 41b254cdf4 change forecast message into an array of messages (elastic/x-pack-elasticsearch#3070)
depends on elastic/machine-learning-cpp#419

Turns the forecast message into an array of messages.

Original commit: elastic/x-pack-elasticsearch@7598342712
2017-11-21 11:47:34 +01:00
Luca Cavanna 941c0a5701 Cross Cluster Search: optionally skip disconnected clusters (elastic/x-pack-elasticsearch#2823)
Original commit: elastic/x-pack-elasticsearch@3b0017df1f
2017-11-21 11:42:39 +01:00
David Roberts f06acdc219 [ML] Improve the way ML jobs are allocated to nodes (elastic/x-pack-elasticsearch#2975)
This change modifies the way ML jobs are assigned to nodes to primarily
base the decision on the estimated memory footprint of the jobs. The
memory footprint comes from the model size stats if the job has been
running long enough, otherwise from the model memory limit. In addition,
an allowance for the program code and stack is added.

If insufficient information is available to base the allocation decision on
memory requirements then the decision falls back to using simple job
counts per node.

relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@b276aedf2f
2017-11-21 09:51:52 +00:00
Lisa Cawley 77803b06c5 [DOCS] Remove http.ssl.enabled step from installation (elastic/x-pack-elasticsearch#3065)
Original commit: elastic/x-pack-elasticsearch@b071fa191f
2017-11-20 11:29:13 -08:00
Lisa Cawley 712e740761 [DOCS] Added sysgenkey command and watcher encryption settings (elastic/x-pack-elasticsearch#3043)
* [DOCS] Added sysgenkey command and watcher settings

* [DOCS] Added data encryption task for Watcher

* [DOCS] Addressed feedback about watcher encryption

Original commit: elastic/x-pack-elasticsearch@edb1fccbfb
2017-11-20 08:44:43 -08:00
lcawley 5ec688a931 [DOCS] Removed beta warning from ML tutorial
Original commit: elastic/x-pack-elasticsearch@38d480fe06
2017-11-20 07:22:59 -08:00
David Kyle 040be7da9d [DOCS][TESTS] Don’t randomise content type for docs tests. (elastic/x-pack-elasticsearch#3058)
Original commit: elastic/x-pack-elasticsearch@7617901ff8
2017-11-20 13:32:14 +00:00
Michael Basnight f8cb1e603b Removed unused Terminal param (elastic/x-pack-elasticsearch#2854)
Relates elastic/elasticsearch#27216

Original commit: elastic/x-pack-elasticsearch@70bdda51ef
2017-11-19 22:33:27 -06:00
Michael Basnight 316da9a970 Move the CLI into its own subproject (elastic/x-pack-elasticsearch#3032)
relates elastic/elasticsearch#27114

Original commit: elastic/x-pack-elasticsearch@70e8488223
2017-11-18 21:43:25 -06:00
Nhat Nguyen 7c9af72d6e Removes the old handler for shard snapshot status
Relates https://github.com/elastic/elasticsearch/pull/27443

Original commit: elastic/x-pack-elasticsearch@7f0021e3f4
2017-11-17 20:14:44 -05:00
lcawley 32d0c1b0c7 [DOCS] Re-enabled code snippet testing
Original commit: elastic/x-pack-elasticsearch@31fd4c3668
2017-11-17 13:40:46 -08:00
lcawley a08ded367c Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@a66f322330
2017-11-17 13:30:20 -08:00
jaymode 34ecd18e76 Remove use of forbidden API URL#getPath
tool. However, this is a forbidden API so this commit replaces it with URI#getPath. Additionally,
the tests fail with a security manager permission error due to the use of Mockito for exception
throwing. This commit still uses Mockito for throwing exceptions but does it differently in a way
that is acceptable by our test security policy.

Original commit: elastic/x-pack-elasticsearch@5e1d45acf8
2017-11-17 14:03:26 -07:00
lcawley a7e0dfb3ec Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@209befd1b1
2017-11-17 12:58:43 -08:00
Albert Zaharovits 5b73c77011 SetupPasswordTool handle url cmd option correctly (elastic/x-pack-elasticsearch#2899)
Fixes bug when the url option had trailing slashes. The URL built
was invalid (consecutive fwd slashes) but the failure errors of
the subsequent requests were ignored.

URL is built correctly from the option spec.
True HTTP errors and Exceptions are logged and the cmd fails.

relates elastic/x-pack-elasticsearch#2778

Original commit: elastic/x-pack-elasticsearch@62b2d94ca0
2017-11-17 20:30:22 +02:00
Nhat Nguyen 418a6632e5 Registers a new handler for shard snapshot status
Relates elastic/x-pack-elasticsearch#27165

Original commit: elastic/x-pack-elasticsearch@11199642bb
2017-11-17 13:13:50 -05:00
lcawley 5dfb35e2a8 Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@f06f403c04
2017-11-17 10:03:54 -08:00