Commit Graph

7211 Commits

Author SHA1 Message Date
jaymode a8b5b138a7 Test: print cluster state xcontent on security index check failures
Original commit: elastic/x-pack-elasticsearch@0ff85f851c
2017-11-22 12:57:10 -07:00
lcawley 79cb3a8a2c [DOCS] Fixed typo
Original commit: elastic/x-pack-elasticsearch@a2da8f644f
2017-11-22 10:02:24 -08:00
Igor Kupczyński 18103fae7f Invalid value in the docs for transport.profiles...client_authentication (elastic/x-pack-elasticsearch#3091)
The 6.x and 6.0 versions of the documentation show

```yml
transport.profiles.client.xpack.security.ssl.client_authentication: no
```

Which results in 
```
2017-11-22T11:13:33,225][ERROR][org.elasticsearch.bootstrap.Bootstrap] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:452) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:392) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:142) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.node.Node.<init>(Node.java:302) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 14 more
Caused by: java.lang.IllegalArgumentException: could not resolve ssl client auth. unknown value [no]
	at org.elasticsearch.xpack.ssl.SSLClientAuth.parse(SSLClientAuth.java:78) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLConfigurationSettings.lambda$null$27(SSLConfigurationSettings.java:183) ~[?:?]
	at org.elasticsearch.common.settings.Setting.get(Setting.java:352) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.settings.Setting.get(Setting.java:346) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.xpack.ssl.SSLConfiguration.<init>(SSLConfiguration.java:80) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService.lambda$loadSSLConfigurations$1(SSLService.java:462) ~[?:?]
	at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_144]
	at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:461) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:87) ~[?:?]
	at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:237) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 14 more
```

I think the valid value there is `none`, so

```yml
transport.profiles.client.xpack.security.ssl.client_authentication: none
```

The tests seem to confirm that https://github.com/elastic/x-pack-elasticsearch/blob/elastic/x-pack-elasticsearch@4860e92d906e046a23aa07b39ee6ef637f011dc1/plugin/src/test/java/org/elasticsearch/xpack/ssl/SSLServiceTests.java#L269

Original commit: elastic/x-pack-elasticsearch@a35b3ac8c9
2017-11-22 17:09:06 +00:00
Nik Everett f97f56ba54 SQL: Throw exceptions on errors (elastic/x-pack-elasticsearch#3066)
Instead of returning "error response" objects and then translating them
into SQL exceptions this just throws the SQL exceptions directly. This
means the CLI catches exceptions and prints out the messages which isn't
ideal if this were hot code but it isn't and this is a much simpler way
of doing things.

Original commit: elastic/x-pack-elasticsearch@08431d3941
2017-11-22 11:22:31 -05:00
Jay Modi 0a683a0e18 Remove InternalClient and InternalSecurityClient (elastic/x-pack-elasticsearch#3054)
This change removes the InternalClient and the InternalSecurityClient. These are replaced with
usage of the ThreadContext and a transient value, `action.origin`, to indicate which component the
request came from. The security code has been updated to look for this value and ensure the
request is executed as the proper user. This work comes from elastic/x-pack-elasticsearch#2808 where @s1monw suggested
that we do this.

While working on this, I came across index template registries and rather than updating them to use
the new method, I replaced the ML one with the template upgrade framework so that we could
remove this template registry. The watcher template registry is still needed as the template must be
updated for rolling upgrades to work (see elastic/x-pack-elasticsearch#2950).

Original commit: elastic/x-pack-elasticsearch@7dbf2f263e
2017-11-22 08:35:18 -07:00
Alexander Reelsen c7a64667d4 Watcher: Properly url encode room names (elastic/x-pack-elasticsearch#2896)
Room names in hipchat were not properly URL encoded, thus room names
with spaces would not work as expected. This fixes all the hipchat
accounts by properly using spaces.

Also the hipchat tests are reenabled, as the IT team gave me new access to hipchat, 
allowing to create a fresh set of oauth tokens for the integration account type.

The HipchatServiceTests have also been converted to XPackSingleNodeTestCase

relates elastic/x-pack-elasticsearch#2371
relates elastic/x-pack-elasticsearch#2429

Original commit: elastic/x-pack-elasticsearch@9f8872f686
2017-11-22 15:50:18 +01:00
Dimitrios Athanasiou 02c83a3b6a [ML][TEST] Add delays between forecasts to ensure id uniqueness
relates elastic/x-pack-elasticsearch#3090

Original commit: elastic/x-pack-elasticsearch@5d6b091607
2017-11-22 12:04:44 +00:00
David Kyle 0dea758022 [ML] Log deprecation warning for jobs with delimited formats (elastic/x-pack-elasticsearch#3092)
Original commit: elastic/x-pack-elasticsearch@bd75fae990
2017-11-22 11:53:08 +00:00
Igor Motov a4915a5714 SQL: remove all remaining NOCOMMITs
relates elastic/x-pack-elasticsearch#2873

Original commit: elastic/x-pack-elasticsearch@68b206efd2
2017-11-21 14:37:59 -05:00
Dimitris Athanasiou 74beb9ca64 [ML] Remove expired forecasts (elastic/x-pack-elasticsearch#3077)
Closes elastic/machine-learning-cpp#322


Original commit: elastic/x-pack-elasticsearch@5249452a86
2017-11-21 17:18:04 +00:00
Jay Modi 4ae1ca5fa5 Security: IndexLifecycleManager provides a consistent view of index state (elastic/x-pack-elasticsearch#3008)
This commit changes the IndexLifecycleManager's handling of variables about an index to only update
all of the values at a single time. Previously, all of the index state variables were volatile
members of the IndexLifecycleManager, which meant we could get an inconsistent view of the index
state. Although rare, this is still incorrect so this change adds a single volatile variable that
holds the state as of the last processed cluster state update.

Additionally, the IndexLifecycleManagerIntegTests were updated to have more concurrency and further
stress this portion of the code and its checks.

relates elastic/x-pack-elasticsearch#2973

Original commit: elastic/x-pack-elasticsearch@5f1552b298
2017-11-21 10:17:08 -07:00
Igor Motov 2fe4da80ad SQL: add filter support in REST action (elastic/x-pack-elasticsearch#3045)
Adds the option to specify an elasticsearch filter in addition to the SQL query by introducing a filter parameter in the REST query which would create a boolean filter if the SQL query generates an elasticsearch query or a constant score query if SQL if the SQL query doesn't generates an elasticsearch query. Usage:

{
   "query": "SELECT * FROM index",
   "filter" : {  "term" : { "tag" : "tech" } }
}

relates elastic/x-pack-elasticsearch#2895

Original commit: elastic/x-pack-elasticsearch@9a73813c7f
2017-11-21 11:40:38 -05:00
Igor Motov 3e14c30aa1 SQL: Add version to cursor serialization (elastic/x-pack-elasticsearch#3064)
This commits also simplifies the serialization mechanism by remove 2 ways to serialize the cursor. Adding the version there was complicating things too much otherwise.

Original commit: elastic/x-pack-elasticsearch@4f2c541e0a
2017-11-21 11:24:27 -05:00
Jay Modi d86e7870da Security: add manage_index_templates to the kibana_system role (elastic/x-pack-elasticsearch#3009)
This commit adds the manage_index_templates permission to the kibana_system role that is used by
the kibana system user. This is needed due to an upcoming feature in kibana where a index template
will be used to create the saved objects index.

relates elastic/x-pack-elasticsearch#2937

Original commit: elastic/x-pack-elasticsearch@85a67c73aa
2017-11-21 08:45:07 -07:00
Nik Everett 0d4a91af50 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@989e27840f
2017-11-21 10:34:23 -05:00
David Roberts 35551859c5 [TEST] Fix side effects of elastic/x-pack-elasticsearch#2975 on build servers with very little RAM
Some of our REST tests open many jobs, and assuming each will use 1GB of
RAM on a single node cluster could fail the test.  The solution is to
explicitly say the test jobs will use very little RAM.

Original commit: elastic/x-pack-elasticsearch@a3fcfc4589
2017-11-21 15:11:28 +00:00
Hendrik Muhs cc66020cf3 [ML-FC] add expires_in parameter and change forecast_start_timestamp to timestamp (elastic/x-pack-elasticsearch#3073)
add expires_in parameter and change forecast_start_timestamp to timestamp

depends on elastic/machine-learning-cpp#421

Original commit: elastic/x-pack-elasticsearch@3a3eebd49c
2017-11-21 15:32:06 +01:00
Simon Willnauer 601222903d X-Pack side of elastic/elasticsearch#27469 (elastic/x-pack-elasticsearch#3071)
Original commit: elastic/x-pack-elasticsearch@99499b6bd6
2017-11-21 15:15:24 +01:00
Dimitris Athanasiou e71b5639de [ML] Rename id to forecast_id in forecast API response (elastic/x-pack-elasticsearch#3074)
Original commit: elastic/x-pack-elasticsearch@c05d9fc602
2017-11-21 13:57:41 +00:00
javanna 61f13b9642 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@d11ddc7a2c
2017-11-21 14:13:24 +01:00
Dimitris Athanasiou 83ca6e8064 [ML] Report errors back to listener in DeleteExpiredDataAction (elastic/x-pack-elasticsearch#3072)
Currently, any errors that occur during the DeleteExpiredDataAction are logged and the deletion proceeds to the next job. The user will get no indication in the response that something went wrong although nothing should really go wrong unless the cluster is messed up.

This commit changes this so that errors are reported back to the action.

Original commit: elastic/x-pack-elasticsearch@489cf03c3e
2017-11-21 12:03:04 +00:00
Dimitris Athanasiou 754623753a [ML] Make it easier to add various ml data removal (elastic/x-pack-elasticsearch#3048)
Original commit: elastic/x-pack-elasticsearch@3e4ac6033b
2017-11-21 11:43:01 +00:00
David Roberts 402852a4ee Update BWC version after backporting to 6.1
Relates elastic/x-pack-elasticsearch#2975

Original commit: elastic/x-pack-elasticsearch@a63c56a019
2017-11-21 10:57:44 +00:00
Hendrik Muhs 41b254cdf4 change forecast message into an array of messages (elastic/x-pack-elasticsearch#3070)
depends on elastic/machine-learning-cpp#419

Turns the forecast message into an array of messages.

Original commit: elastic/x-pack-elasticsearch@7598342712
2017-11-21 11:47:34 +01:00
Luca Cavanna 941c0a5701 Cross Cluster Search: optionally skip disconnected clusters (elastic/x-pack-elasticsearch#2823)
Original commit: elastic/x-pack-elasticsearch@3b0017df1f
2017-11-21 11:42:39 +01:00
Igor Motov 318621cdc1 SQL: Move all SQL projects into org.elasticsearch.sql package (elastic/x-pack-elasticsearch#3069)
Elasticsearch plugins are traditionally using org.elasticsearch.plugin group, clients are using org.elasticsearch.client, so it makes sense to move all sql pieces and deliverables into org.elasticsearch.sql group. Otherwise, users might get really confused about an artifact named `server` in the group `org.elasticsearch`.

Original commit: elastic/x-pack-elasticsearch@c3a4042040
2017-11-21 10:53:40 +01:00
David Roberts f06acdc219 [ML] Improve the way ML jobs are allocated to nodes (elastic/x-pack-elasticsearch#2975)
This change modifies the way ML jobs are assigned to nodes to primarily
base the decision on the estimated memory footprint of the jobs. The
memory footprint comes from the model size stats if the job has been
running long enough, otherwise from the model memory limit. In addition,
an allowance for the program code and stack is added.

If insufficient information is available to base the allocation decision on
memory requirements then the decision falls back to using simple job
counts per node.

relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@b276aedf2f
2017-11-21 09:51:52 +00:00
Igor Motov 0d398b19ce Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@6b82e2c12e
2017-11-20 17:41:43 -05:00
Lisa Cawley 77803b06c5 [DOCS] Remove http.ssl.enabled step from installation (elastic/x-pack-elasticsearch#3065)
Original commit: elastic/x-pack-elasticsearch@b071fa191f
2017-11-20 11:29:13 -08:00
Lisa Cawley 712e740761 [DOCS] Added sysgenkey command and watcher encryption settings (elastic/x-pack-elasticsearch#3043)
* [DOCS] Added sysgenkey command and watcher settings

* [DOCS] Added data encryption task for Watcher

* [DOCS] Addressed feedback about watcher encryption

Original commit: elastic/x-pack-elasticsearch@edb1fccbfb
2017-11-20 08:44:43 -08:00
lcawley 5ec688a931 [DOCS] Removed beta warning from ML tutorial
Original commit: elastic/x-pack-elasticsearch@38d480fe06
2017-11-20 07:22:59 -08:00
David Kyle 040be7da9d [DOCS][TESTS] Don’t randomise content type for docs tests. (elastic/x-pack-elasticsearch#3058)
Original commit: elastic/x-pack-elasticsearch@7617901ff8
2017-11-20 13:32:14 +00:00
Michael Basnight f8cb1e603b Removed unused Terminal param (elastic/x-pack-elasticsearch#2854)
Relates elastic/elasticsearch#27216

Original commit: elastic/x-pack-elasticsearch@70bdda51ef
2017-11-19 22:33:27 -06:00
Michael Basnight 316da9a970 Move the CLI into its own subproject (elastic/x-pack-elasticsearch#3032)
relates elastic/elasticsearch#27114

Original commit: elastic/x-pack-elasticsearch@70e8488223
2017-11-18 21:43:25 -06:00
Nhat Nguyen 7c9af72d6e Removes the old handler for shard snapshot status
Relates https://github.com/elastic/elasticsearch/pull/27443

Original commit: elastic/x-pack-elasticsearch@7f0021e3f4
2017-11-17 20:14:44 -05:00
lcawley 32d0c1b0c7 [DOCS] Re-enabled code snippet testing
Original commit: elastic/x-pack-elasticsearch@31fd4c3668
2017-11-17 13:40:46 -08:00
lcawley a08ded367c Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@a66f322330
2017-11-17 13:30:20 -08:00
jaymode 34ecd18e76 Remove use of forbidden API URL#getPath
tool. However, this is a forbidden API so this commit replaces it with URI#getPath. Additionally,
the tests fail with a security manager permission error due to the use of Mockito for exception
throwing. This commit still uses Mockito for throwing exceptions but does it differently in a way
that is acceptable by our test security policy.

Original commit: elastic/x-pack-elasticsearch@5e1d45acf8
2017-11-17 14:03:26 -07:00
lcawley a7e0dfb3ec Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@209befd1b1
2017-11-17 12:58:43 -08:00
Igor Motov d011247970 SQL: Fix testJdbcActionLicense test
The testJdbcActionLicense test broke because we changed the error serialization mechanism in elastic/x-pack-elasticsearch#3034. This commit updates that check to capture an exception instead of checking for ErrorResponse.

Original commit: elastic/x-pack-elasticsearch@17c41426d0
2017-11-17 15:37:02 -05:00
Igor Motov ffcd54de77 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@e67ceb1d1e
2017-11-17 13:33:28 -05:00
Albert Zaharovits 5b73c77011 SetupPasswordTool handle url cmd option correctly (elastic/x-pack-elasticsearch#2899)
Fixes bug when the url option had trailing slashes. The URL built
was invalid (consecutive fwd slashes) but the failure errors of
the subsequent requests were ignored.

URL is built correctly from the option spec.
True HTTP errors and Exceptions are logged and the cmd fails.

relates elastic/x-pack-elasticsearch#2778

Original commit: elastic/x-pack-elasticsearch@62b2d94ca0
2017-11-17 20:30:22 +02:00
Nhat Nguyen 418a6632e5 Registers a new handler for shard snapshot status
Relates elastic/x-pack-elasticsearch#27165

Original commit: elastic/x-pack-elasticsearch@11199642bb
2017-11-17 13:13:50 -05:00
lcawley 5dfb35e2a8 Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@f06f403c04
2017-11-17 10:03:54 -08:00
Nik Everett 87a3e93fae SQL: Improve error message for huge error responses (elastic/x-pack-elasticsearch#3049)
If the user has a funky proxy or something that throws back a massive
error message to the CLI or JDBC then return a sensible error message.

Original commit: elastic/x-pack-elasticsearch@94219969ea
2017-11-17 12:29:07 -05:00
Nik Everett fce5b494be SQL: Stop serializing errors for CLI and JDBC (elastic/x-pack-elasticsearch#3034)
Now that we can parse Elasticsearch's standard error messages in the CLI
and JDBC client we can just let those standard error messages bubble out
of Elasticsearch rather than catch and encode them.

In a followup we can remove the encoding entirely.

Original commit: elastic/x-pack-elasticsearch@bad043b6f7
2017-11-17 12:16:37 -05:00
Nik Everett b8e082107f Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@bbc72c0368
2017-11-17 12:05:47 -05:00
David Roberts 6741a62e80 [ML] Adjust memory limit test to account for greater accuracy (elastic/x-pack-elasticsearch#3047)
Due to elastic/machine-learning-cpp#409 the ML C++ code now instruments
memory more accurately.  This change modifies the expectations of the Java
integration test to account for the change.

Original commit: elastic/x-pack-elasticsearch@2ed7a75af4
2017-11-17 17:02:19 +00:00
Alexander Reelsen 0f97e28074 Watcher: Further preparations for source repo split (elastic/x-pack-elasticsearch#3006)
Changes to further prepare for feature split with regards to watcher:

- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
  used there
- The creation of the http client/notification classes have been moved
  into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
2017-11-17 17:05:07 +01:00
Alexander Reelsen 3d5fb54522 Watcher: Create templates on nodes newer than the master (elastic/x-pack-elasticsearch#2950)
This problem was introduced due to distributed watch execution.

When a node newer than the master node joins the cluster and gets a
.watches shard assigned it is supposed to start watcher. However
when a new version of the watch history template is part of that new
node (and we might increase that version anytime), this template does
not get installed, because only the master node is updating watcher
templates.

This commit checks if the local node version is higher than the master
node version and then also puts missing templates.

Currently this is done for all watcher templates, not only the watcher history.

relates elastic/x-pack-elasticsearch#2944

Original commit: elastic/x-pack-elasticsearch@4960231ea7
2017-11-17 16:56:29 +01:00