From 7ae8f1cd9b04675a39d3c2eaa94ece36b7bd88a4 Mon Sep 17 00:00:00 2001 From: lcawley Date: Thu, 16 Nov 2017 15:40:51 -0800 Subject: [PATCH 01/11] [DOCS] Added sysgenkey command and watcher settings Original commit: elastic/x-pack-elasticsearch@8e0b2027a69c88486036b82644cde0d95d8e3823 --- docs/en/commands/index.asciidoc | 2 + docs/en/commands/syskeygen.asciidoc | 60 +++++++++++++++++++ .../settings/notification-settings.asciidoc | 28 +++++++-- 3 files changed, 86 insertions(+), 4 deletions(-) create mode 100644 docs/en/commands/syskeygen.asciidoc diff --git a/docs/en/commands/index.asciidoc b/docs/en/commands/index.asciidoc index 69f279f90ff..69811a9fc8d 100644 --- a/docs/en/commands/index.asciidoc +++ b/docs/en/commands/index.asciidoc @@ -9,10 +9,12 @@ * <> * <> +* <> * <> -- include::certgen.asciidoc[] include::setup-passwords.asciidoc[] +include::syskeygen.asciidoc[] include::users-command.asciidoc[] diff --git a/docs/en/commands/syskeygen.asciidoc b/docs/en/commands/syskeygen.asciidoc new file mode 100644 index 00000000000..be8708ecd37 --- /dev/null +++ b/docs/en/commands/syskeygen.asciidoc @@ -0,0 +1,60 @@ +[role="xpack"] +[[syskeygen]] +== syskeygen + +The `syskeygen` command creates a system key file in +`CONFIG_DIR/x-pack/system_key`. You can then copy the generated system key to +the rest of the nodes in the cluster. + +//TBD: Why? What's its purpose? + +IMPORTANT: The system key is a symmetric key, so the same key must be on every +node in the cluster. You must also ensure that the file permissions are set +such that the key can be read by the user that {es} runs as. + +[float] +=== Synopsis + +[source,shell] +-------------------------------------------------- +bin/x-pack/syskeygen +[-E ] [-h, --help] +([-s, --silent] | [-v, --verbose]) +-------------------------------------------------- + +[float] +=== Description + +Encryption of sensitive data requires a key. For example, if you enable the +<>, +//TBD: Can we clarify what type of sensitive data is encrypted when this is enabled? +you must also create a system key file and place it in the secure setting store. +After you run the `syskeygen` command, run the +`elasticsearch-keystore add-file xpack.watcher.encryption_key ` +command. Finally, remove the `system_key` file from the filesystem. Repeat these +steps on every node in the cluster. + +//TBD: Are there other scenarios where you could use this system file? + +[float] +=== Parameters + +`-E `:: Configures a setting. + +`-h, --help`:: Returns all of the command parameters. + +`-s, --silent`:: Shows minimal output. + +`-v, --verbose`:: Shows verbose output. + +//// +[float] +=== Examples + +//TBD: What's an example of a -E value that we might use in this command? + +[source, sh] +-------------------------------------------------- +bin/x-pack/syskeygen +-------------------------------------------------- +//// diff --git a/docs/en/settings/notification-settings.asciidoc b/docs/en/settings/notification-settings.asciidoc index 9432cf96b63..65d536e7e7c 100644 --- a/docs/en/settings/notification-settings.asciidoc +++ b/docs/en/settings/notification-settings.asciidoc @@ -5,10 +5,16 @@ {watcher} Settings ++++ -You configure `xpack.notification` settings in `elasticsearch.yml` to -send set up {watcher} and send notifications via <>, -<>, <>, and <>. +You configure {watcher} settings to set up {watcher} and send notifications via +<>, +<>, +<>, and +<>. + +All of these settings can be added to the `elasticsearch.yml` configuration file, +with the exception of the secure settings, which you add to the {es} keystore. +For more information about creating and updating the {es} keystore, see +<>. [float] [[general-notification-settings]] @@ -16,6 +22,20 @@ Slack>>, and <>. `xpack.watcher.enabled`:: Set to `false` to disable {watcher} on the node. +`xpack.watcher.encrypt_sensitive_data` (<>):: +Set to `true` to encrypt sensitive data. If this setting is enabled, you +must also specify the `xpack.watcher.encryption_key` setting. +//TBD: Can we be more specific about what type of watcher data it's encrypting? + +`xpack.watcher.encryption_key` (<>):: +Specifies the path to a file that contains a key for encrypting sensitive data. +If `xpack.watcher.encrypt_sensitive_data` is set to `true`, this setting is +required. For example, to generate the key file, run the <> tool. +Then import the key into the secure setting store with the +`elasticsearch-keystore add-file xpack.watcher.encryption_key ` +command. Finally, remove the system_key file from the filesystem. Repeat this on +every node in the cluster. + `xpack.watcher.history.cleaner_service.enabled`:: Set to `false` (default) to disable the cleaner service, which removes previous versions of {watcher} indices (for example, .watcher-history*) when it From 5b73c77011f6ad097ce36f784df963afdd014f31 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 6 Nov 2017 20:41:53 +0200 Subject: [PATCH 02/11] SetupPasswordTool handle url cmd option correctly (elastic/x-pack-elasticsearch#2899) Fixes bug when the url option had trailing slashes. The URL built was invalid (consecutive fwd slashes) but the failure errors of the subsequent requests were ignored. URL is built correctly from the option spec. True HTTP errors and Exceptions are logged and the cmd fails. relates elastic/x-pack-elasticsearch#2778 Original commit: elastic/x-pack-elasticsearch@62b2d94ca076061de3dd7505fedfdba803e18385 --- .../esnative/tool/CommandLineHttpClient.java | 1 - .../esnative/tool/SetupPasswordTool.java | 100 ++++++++++++------ .../esnative/tool/SetupPasswordToolTests.java | 77 ++++++++++---- 3 files changed, 127 insertions(+), 51 deletions(-) diff --git a/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/CommandLineHttpClient.java b/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/CommandLineHttpClient.java index 0ed59110fd8..2b7acc14954 100644 --- a/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/CommandLineHttpClient.java +++ b/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/CommandLineHttpClient.java @@ -5,7 +5,6 @@ */ package org.elasticsearch.xpack.security.authc.esnative.tool; -import org.bouncycastle.util.io.Streams; import org.elasticsearch.common.CheckedConsumer; import org.elasticsearch.common.CheckedSupplier; import org.elasticsearch.common.Strings; diff --git a/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordTool.java b/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordTool.java index 764f4531ff5..05cdfac7dc4 100644 --- a/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordTool.java +++ b/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordTool.java @@ -7,8 +7,8 @@ package org.elasticsearch.xpack.security.authc.esnative.tool; import javax.net.ssl.SSLException; import java.io.IOException; +import java.io.InputStream; import java.net.HttpURLConnection; -import java.net.SocketException; import java.net.URL; import java.nio.charset.StandardCharsets; import java.security.SecureRandom; @@ -21,6 +21,7 @@ import joptsimple.OptionParser; import joptsimple.OptionSet; import joptsimple.OptionSpec; import org.bouncycastle.util.io.Streams; +import org.elasticsearch.ExceptionsHelper; import org.elasticsearch.cli.EnvironmentAwareCommand; import org.elasticsearch.cli.ExitCodes; import org.elasticsearch.cli.MultiCommand; @@ -113,7 +114,7 @@ public class SetupPasswordTool extends MultiCommand { checkElasticKeystorePasswordValid(terminal, env); if (shouldPrompt) { - terminal.println("Initiating the setup of reserved user " + String.join(",", USERS) + " passwords."); + terminal.println("Initiating the setup of passwords for reserved users " + String.join(",", USERS) + "."); terminal.println("The passwords will be randomly generated and printed to the console."); boolean shouldContinue = terminal.promptYesNo("Please confirm that you would like to continue", false); terminal.println("\n"); @@ -124,7 +125,7 @@ public class SetupPasswordTool extends MultiCommand { SecureRandom secureRandom = new SecureRandom(); changePasswords((user) -> generatePassword(secureRandom, user), - (user, password) -> changedPasswordCallback(terminal, user, password)); + (user, password) -> changedPasswordCallback(terminal, user, password), terminal); } private SecureString generatePassword(SecureRandom secureRandom, String user) { @@ -158,7 +159,7 @@ public class SetupPasswordTool extends MultiCommand { checkElasticKeystorePasswordValid(terminal, env); if (shouldPrompt) { - terminal.println("Initiating the setup of reserved user " + String.join(",", USERS) + " passwords."); + terminal.println("Initiating the setup of passwords for reserved users " + String.join(",", USERS) + "."); terminal.println("You will be prompted to enter passwords as the process progresses."); boolean shouldContinue = terminal.promptYesNo("Please confirm that you would like to continue", false); terminal.println("\n"); @@ -168,7 +169,7 @@ public class SetupPasswordTool extends MultiCommand { } changePasswords(user -> promptForPassword(terminal, user), - (user, password) -> changedPasswordCallback(terminal, user, password)); + (user, password) -> changedPasswordCallback(terminal, user, password), terminal); } private SecureString promptForPassword(Terminal terminal, String user) throws UserException { @@ -212,7 +213,7 @@ public class SetupPasswordTool extends MultiCommand { private String elasticUser = ElasticUser.NAME; private SecureString elasticUserPassword; - private String url; + private URL url; SetupCommand(String description) { super(description); @@ -223,7 +224,7 @@ public class SetupPasswordTool extends MultiCommand { client = clientFunction.apply(env); try (KeyStoreWrapper keyStore = keyStoreFunction.apply(env)) { String providedUrl = urlOption.value(options); - url = providedUrl == null ? client.getDefaultURL() : providedUrl; + url = new URL(providedUrl == null ? client.getDefaultURL() : providedUrl); setShouldPrompt(options); // TODO: We currently do not support keystore passwords @@ -234,7 +235,7 @@ public class SetupPasswordTool extends MultiCommand { } private void setParser() { - urlOption = parser.acceptsAll(Arrays.asList("u", "url"), "The url for the change password request.").withOptionalArg(); + urlOption = parser.acceptsAll(Arrays.asList("u", "url"), "The url for the change password request.").withRequiredArg(); noPromptOption = parser.acceptsAll(Arrays.asList("b", "batch"), "If enabled, run the change password process without prompting the user.").withOptionalArg(); } @@ -257,14 +258,12 @@ public class SetupPasswordTool extends MultiCommand { * where to write verbose info. */ void checkElasticKeystorePasswordValid(Terminal terminal, Environment env) throws Exception { - URL route = new URL(url + "/_xpack/security/_authenticate?pretty"); + URL route = new URL(url, (url.getPath() + "/_xpack/security/_authenticate").replaceAll("/+", "/") + "?pretty"); + terminal.println(Verbosity.VERBOSE, ""); + terminal.println(Verbosity.VERBOSE, "Testing if bootstrap password is valid for " + route.toString()); try { - terminal.println(Verbosity.VERBOSE, ""); - terminal.println(Verbosity.VERBOSE, "Testing if bootstrap password is valid for " + route.toString()); - int httpCode = client.postURL("GET", route, elasticUser, elasticUserPassword, () -> null, is -> { - byte[] bytes = Streams.readAll(is); - terminal.println(Verbosity.VERBOSE, new String(bytes, StandardCharsets.UTF_8)); - }); + final int httpCode = client.postURL("GET", route, elasticUser, elasticUserPassword, () -> null, + is -> verboseLogResponse(is, terminal)); // keystore password is not valid if (httpCode == HttpURLConnection.HTTP_UNAUTHORIZED) { terminal.println(""); @@ -275,23 +274,33 @@ public class SetupPasswordTool extends MultiCommand { terminal.println(" This tool used the keystore at " + KeyStoreWrapper.keystorePath(env.configFile())); terminal.println(""); throw new UserException(ExitCodes.CONFIG, "Failed to verify bootstrap password"); + } else if (httpCode != HttpURLConnection.HTTP_OK) { + terminal.println(""); + terminal.println("Unexpected response code [" + httpCode + "] from calling GET " + route.toString()); + terminal.println("Possible causes include:"); + terminal.println(" * The relative path of the URL is incorrect. Is there a proxy in-between?"); + terminal.println(" * The protocol (http/https) does not match the port."); + terminal.println(" * Is this really an Elasticsearch server?"); + terminal.println(""); + throw new UserException(ExitCodes.CONFIG, "Uknown error"); } - } catch (SocketException e) { - terminal.println(""); - terminal.println("Cannot connect to elasticsearch node."); - e.printStackTrace(terminal.getWriter()); - terminal.println(""); - throw new UserException(ExitCodes.CONFIG, - "Failed to connect to elasticsearch at " + route.toString() + ". Is the URL correct and elasticsearch running?", e); } catch (SSLException e) { terminal.println(""); terminal.println("SSL connection to " + route.toString() + " failed: " + e.getMessage()); terminal.println("Please check the elasticsearch SSL settings under " + CommandLineHttpClient.HTTP_SSL_SETTING); - terminal.println(""); - e.printStackTrace(terminal.getWriter()); + terminal.println(Verbosity.VERBOSE, ""); + terminal.println(Verbosity.VERBOSE, ExceptionsHelper.stackTrace(e)); terminal.println(""); throw new UserException(ExitCodes.CONFIG, "Failed to establish SSL connection to elasticsearch at " + route.toString() + ". ", e); + } catch (IOException e) { + terminal.println(""); + terminal.println("Connection failure to: " + route.toString() + " failed: " + e.getMessage()); + terminal.println(Verbosity.VERBOSE, ""); + terminal.println(Verbosity.VERBOSE, ExceptionsHelper.stackTrace(e)); + terminal.println(""); + throw new UserException(ExitCodes.CONFIG, "Failed to connect to elasticsearch at " + + route.toString() + ". Is the URL correct and elasticsearch running?", e); } } @@ -303,12 +312,14 @@ public class SetupPasswordTool extends MultiCommand { * @param password * the new password of the user. */ - private void changeUserPassword(String user, SecureString password) throws Exception { - URL route = new URL(url + "/_xpack/security/user/" + user + "/_password"); + private void changeUserPassword(String user, SecureString password, Terminal terminal) throws Exception { + URL route = new URL(url, (url.getPath() + "/_xpack/security/user/" + user + "/_password").replaceAll("/+", "/") + "?pretty"); + terminal.println(Verbosity.VERBOSE, ""); + terminal.println(Verbosity.VERBOSE, "Trying user password change call " + route.toString()); try { // supplier should own his resources SecureString supplierPassword = password.clone(); - client.postURL("PUT", route, elasticUser, elasticUserPassword, () -> { + final int httpCode = client.postURL("PUT", route, elasticUser, elasticUserPassword, () -> { try { XContentBuilder xContentBuilder = JsonXContent.contentBuilder(); xContentBuilder.startObject().field("password", supplierPassword.toString()).endObject(); @@ -316,9 +327,24 @@ public class SetupPasswordTool extends MultiCommand { } finally { supplierPassword.close(); } - }, is -> { - }); + }, is -> verboseLogResponse(is, terminal)); + if (httpCode != HttpURLConnection.HTTP_OK) { + terminal.println(""); + terminal.println("Unexpected response code [" + httpCode + "] from calling PUT " + route.toString()); + terminal.println("Possible next steps:"); + terminal.println("* Try running this tool again."); + terminal.println("* Check the elasticsearch logs for additional error details."); + terminal.println("* Use the change password API manually. "); + terminal.println(""); + throw new UserException(ExitCodes.TEMP_FAILURE, + "Failed to set password for user [" + user + "]."); + } } catch (IOException e) { + terminal.println(""); + terminal.println("Connection failure to: " + route.toString() + " failed: " + e.getMessage()); + terminal.println(Verbosity.VERBOSE, ""); + terminal.println(Verbosity.VERBOSE, ExceptionsHelper.stackTrace(e)); + terminal.println(""); throw new UserException(ExitCodes.TEMP_FAILURE, "Failed to set password for user [" + user + "].", e); } } @@ -333,7 +359,7 @@ public class SetupPasswordTool extends MultiCommand { * Callback for each successful operation */ void changePasswords(CheckedFunction passwordFn, - CheckedBiConsumer successCallback) throws Exception { + CheckedBiConsumer successCallback, Terminal terminal) throws Exception { Map passwordsMap = new HashMap<>(USERS.size()); try { for (String user : USERS) { @@ -350,17 +376,27 @@ public class SetupPasswordTool extends MultiCommand { superUserEntry = entry; continue; } - changeUserPassword(entry.getKey(), entry.getValue()); + changeUserPassword(entry.getKey(), entry.getValue(), terminal); successCallback.accept(entry.getKey(), entry.getValue()); } // change elastic superuser if (superUserEntry != null) { - changeUserPassword(superUserEntry.getKey(), superUserEntry.getValue()); + changeUserPassword(superUserEntry.getKey(), superUserEntry.getValue(), terminal); successCallback.accept(superUserEntry.getKey(), superUserEntry.getValue()); } } finally { passwordsMap.forEach((user, pass) -> pass.close()); } } + + private void verboseLogResponse(InputStream is, Terminal terminal) throws IOException { + if (is != null) { + byte[] bytes = Streams.readAll(is); + terminal.println(Verbosity.VERBOSE, new String(bytes, StandardCharsets.UTF_8)); + } else { + terminal.println(Verbosity.VERBOSE, ""); + } + } } + } diff --git a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java index 9afab12c42d..ac5a3974024 100644 --- a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java +++ b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java @@ -28,9 +28,9 @@ import org.junit.Before; import org.mockito.ArgumentCaptor; import org.mockito.InOrder; import org.mockito.Mockito; - import java.io.IOException; import java.net.HttpURLConnection; +import java.net.MalformedURLException; import java.net.URL; import java.util.ArrayList; import java.util.Arrays; @@ -38,7 +38,7 @@ import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Map; - +import javax.net.ssl.SSLException; import static org.mockito.Matchers.anyString; import static org.mockito.Matchers.eq; import static org.mockito.Matchers.any; @@ -124,26 +124,31 @@ public class SetupPasswordToolTests extends CommandTestCase { } public void testAutoSetup() throws Exception { - String url = httpClient.getDefaultURL(); - execute("auto", pathHomeParameter, "-b", "true"); + URL url = new URL(httpClient.getDefaultURL()); + if (randomBoolean()) { + execute("auto", pathHomeParameter, "-b", "true"); + } else { + terminal.addTextInput("Y"); + execute("auto", pathHomeParameter); + } verify(keyStore).decrypt(new char[0]); InOrder inOrder = Mockito.inOrder(httpClient); - URL checkUrl = new URL(url + "/_xpack/security/_authenticate?pretty"); + URL checkUrl = checkURL(url); inOrder.verify(httpClient).postURL(eq("GET"), eq(checkUrl), eq(ElasticUser.NAME), eq(bootstrapPassword), any(CheckedSupplier.class), any(CheckedConsumer.class)); for (String user : usersInSetOrder) { - URL urlWithRoute = new URL(url + "/_xpack/security/user/" + user + "/_password"); + URL urlWithRoute = passwdURL(url, user); inOrder.verify(httpClient).postURL(eq("PUT"), eq(urlWithRoute), eq(ElasticUser.NAME), eq(bootstrapPassword), any(CheckedSupplier.class), any(CheckedConsumer.class)); } - } public void testAuthnFail() throws Exception { - URL authnURL = new URL(httpClient.getDefaultURL() + "/_xpack/security/_authenticate?pretty"); + URL url = new URL(httpClient.getDefaultURL()); + URL authnURL = checkURL(url); when(httpClient.postURL(eq("GET"), eq(authnURL), eq(ElasticUser.NAME), any(SecureString.class), any(CheckedSupplier.class), any(CheckedConsumer.class))).thenReturn(HttpURLConnection.HTTP_UNAUTHORIZED); @@ -153,38 +158,66 @@ public class SetupPasswordToolTests extends CommandTestCase { } catch (UserException e) { assertEquals(ExitCodes.CONFIG, e.exitCode); } + } + public void testWrongServer() throws Exception { + URL url = new URL(httpClient.getDefaultURL()); + URL authnURL = checkURL(url); + when(httpClient.postURL(eq("GET"), eq(authnURL), eq(ElasticUser.NAME), any(SecureString.class), any(CheckedSupplier.class), + any(CheckedConsumer.class))).thenThrow(randomFrom(IOException.class, SSLException.class)); + + try { + execute(randomBoolean() ? "auto" : "interactive", pathHomeParameter); + fail("Should have thrown exception"); + } catch (UserException e) { + assertEquals(ExitCodes.CONFIG, e.exitCode); + } } public void testUrlOption() throws Exception { - String url = "http://localhost:9202"; - execute("auto", pathHomeParameter, "-u", url, "-b"); + URL url = new URL("http://localhost:9202" + randomFrom("", "/", "//", "/smth", "//smth/", "//x//x/")); + execute("auto", pathHomeParameter, "-u", url.toString(), "-b"); InOrder inOrder = Mockito.inOrder(httpClient); - URL checkUrl = new URL(url + "/_xpack/security/_authenticate?pretty"); + URL checkUrl = checkURL(url); inOrder.verify(httpClient).postURL(eq("GET"), eq(checkUrl), eq(ElasticUser.NAME), eq(bootstrapPassword), any(CheckedSupplier.class), any(CheckedConsumer.class)); for (String user : usersInSetOrder) { - URL urlWithRoute = new URL(url + "/_xpack/security/user/" + user + "/_password"); + URL urlWithRoute = passwdURL(url, user); inOrder.verify(httpClient).postURL(eq("PUT"), eq(urlWithRoute), eq(ElasticUser.NAME), eq(bootstrapPassword), any(CheckedSupplier.class), any(CheckedConsumer.class)); } } + public void testSetUserPassFail() throws Exception { + URL url = new URL(httpClient.getDefaultURL()); + String userToFail = randomFrom(SetupPasswordTool.USERS); + URL userToFailURL = passwdURL(url, userToFail); + + when(httpClient.postURL(eq("PUT"), eq(userToFailURL), anyString(), any(SecureString.class), any(CheckedSupplier.class), + any(CheckedConsumer.class))).thenThrow(IOException.class); + try { + execute(randomBoolean() ? "auto" : "interactive", pathHomeParameter, "-b"); + fail("Should have thrown exception"); + } catch (UserException e) { + assertEquals(ExitCodes.TEMP_FAILURE, e.exitCode); + } + } + public void testInteractiveSetup() throws Exception { - String url = httpClient.getDefaultURL(); + URL url = new URL(httpClient.getDefaultURL()); terminal.addTextInput("Y"); execute("interactive", pathHomeParameter); InOrder inOrder = Mockito.inOrder(httpClient); - URL checkUrl = new URL(url + "/_xpack/security/_authenticate?pretty"); + URL checkUrl = checkURL(url); inOrder.verify(httpClient).postURL(eq("GET"), eq(checkUrl), eq(ElasticUser.NAME), eq(bootstrapPassword), any(CheckedSupplier.class), any(CheckedConsumer.class)); for (String user : usersInSetOrder) { - URL urlWithRoute = new URL(url + "/_xpack/security/user/" + user + "/_password"); + URL urlWithRoute = passwdURL(url, user); ArgumentCaptor> passwordCaptor = ArgumentCaptor.forClass((Class) CheckedSupplier.class); inOrder.verify(httpClient).postURL(eq("PUT"), eq(urlWithRoute), eq(ElasticUser.NAME), eq(bootstrapPassword), passwordCaptor.capture(), any(CheckedConsumer.class)); @@ -193,7 +226,7 @@ public class SetupPasswordToolTests extends CommandTestCase { } public void testInteractivePasswordsFatFingers() throws Exception { - String url = httpClient.getDefaultURL(); + URL url = new URL(httpClient.getDefaultURL()); terminal.reset(); terminal.addTextInput("Y"); @@ -218,11 +251,11 @@ public class SetupPasswordToolTests extends CommandTestCase { InOrder inOrder = Mockito.inOrder(httpClient); - URL checkUrl = new URL(url + "/_xpack/security/_authenticate?pretty"); + URL checkUrl = checkURL(url); inOrder.verify(httpClient).postURL(eq("GET"), eq(checkUrl), eq(ElasticUser.NAME), eq(bootstrapPassword), any(CheckedSupplier.class), any(CheckedConsumer.class)); for (String user : usersInSetOrder) { - URL urlWithRoute = new URL(url + "/_xpack/security/user/" + user + "/_password"); + URL urlWithRoute = passwdURL(url, user); ArgumentCaptor> passwordCaptor = ArgumentCaptor.forClass((Class) CheckedSupplier.class); inOrder.verify(httpClient).postURL(eq("PUT"), eq(urlWithRoute), eq(ElasticUser.NAME), eq(bootstrapPassword), passwordCaptor.capture(), any(CheckedConsumer.class)); @@ -243,4 +276,12 @@ public class SetupPasswordToolTests extends CommandTestCase { } throw new RuntimeException("Did not properly parse password."); } + + private URL checkURL(URL url) throws MalformedURLException { + return new URL(url, (url.getPath() + "/_xpack/security/_authenticate").replaceAll("/+", "/") + "?pretty"); + } + + private URL passwdURL(URL url, String user) throws MalformedURLException { + return new URL(url, (url.getPath() + "/_xpack/security/user/" + user + "/_password").replaceAll("/+", "/") + "?pretty"); + } } From 34ecd18e76f36b0cd7502ffd3514e6fe9ed6485a Mon Sep 17 00:00:00 2001 From: jaymode Date: Fri, 17 Nov 2017 14:03:26 -0700 Subject: [PATCH 03/11] Remove use of forbidden API URL#getPath tool. However, this is a forbidden API so this commit replaces it with URI#getPath. Additionally, the tests fail with a security manager permission error due to the use of Mockito for exception throwing. This commit still uses Mockito for throwing exceptions but does it differently in a way that is acceptable by our test security policy. Original commit: elastic/x-pack-elasticsearch@5e1d45acf89797f66c6671a089a98ee0ba4ca521 --- .../authc/esnative/tool/SetupPasswordTool.java | 5 +++-- .../esnative/tool/SetupPasswordToolTests.java | 18 ++++++++++-------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordTool.java b/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordTool.java index 05cdfac7dc4..2cea16fee78 100644 --- a/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordTool.java +++ b/plugin/src/main/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordTool.java @@ -258,7 +258,7 @@ public class SetupPasswordTool extends MultiCommand { * where to write verbose info. */ void checkElasticKeystorePasswordValid(Terminal terminal, Environment env) throws Exception { - URL route = new URL(url, (url.getPath() + "/_xpack/security/_authenticate").replaceAll("/+", "/") + "?pretty"); + URL route = new URL(url, (url.toURI().getPath() + "/_xpack/security/_authenticate").replaceAll("/+", "/") + "?pretty"); terminal.println(Verbosity.VERBOSE, ""); terminal.println(Verbosity.VERBOSE, "Testing if bootstrap password is valid for " + route.toString()); try { @@ -313,7 +313,8 @@ public class SetupPasswordTool extends MultiCommand { * the new password of the user. */ private void changeUserPassword(String user, SecureString password, Terminal terminal) throws Exception { - URL route = new URL(url, (url.getPath() + "/_xpack/security/user/" + user + "/_password").replaceAll("/+", "/") + "?pretty"); + URL route = new URL(url, (url.toURI().getPath() + "/_xpack/security/user/" + user + "/_password").replaceAll("/+", "/") + + "?pretty"); terminal.println(Verbosity.VERBOSE, ""); terminal.println(Verbosity.VERBOSE, "Trying user password change call " + route.toString()); try { diff --git a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java index ac5a3974024..5327dc381f0 100644 --- a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java +++ b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java @@ -31,6 +31,7 @@ import org.mockito.Mockito; import java.io.IOException; import java.net.HttpURLConnection; import java.net.MalformedURLException; +import java.net.URISyntaxException; import java.net.URL; import java.util.ArrayList; import java.util.Arrays; @@ -42,6 +43,7 @@ import javax.net.ssl.SSLException; import static org.mockito.Matchers.anyString; import static org.mockito.Matchers.eq; import static org.mockito.Matchers.any; +import static org.mockito.Mockito.doThrow; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; @@ -163,8 +165,8 @@ public class SetupPasswordToolTests extends CommandTestCase { public void testWrongServer() throws Exception { URL url = new URL(httpClient.getDefaultURL()); URL authnURL = checkURL(url); - when(httpClient.postURL(eq("GET"), eq(authnURL), eq(ElasticUser.NAME), any(SecureString.class), any(CheckedSupplier.class), - any(CheckedConsumer.class))).thenThrow(randomFrom(IOException.class, SSLException.class)); + doThrow(randomFrom(new IOException(), new SSLException(""))).when(httpClient).postURL(eq("GET"), eq(authnURL), eq(ElasticUser.NAME), + any(SecureString.class), any(CheckedSupplier.class), any(CheckedConsumer.class)); try { execute(randomBoolean() ? "auto" : "interactive", pathHomeParameter); @@ -195,8 +197,8 @@ public class SetupPasswordToolTests extends CommandTestCase { String userToFail = randomFrom(SetupPasswordTool.USERS); URL userToFailURL = passwdURL(url, userToFail); - when(httpClient.postURL(eq("PUT"), eq(userToFailURL), anyString(), any(SecureString.class), any(CheckedSupplier.class), - any(CheckedConsumer.class))).thenThrow(IOException.class); + doThrow(new IOException()).when(httpClient).postURL(eq("PUT"), eq(userToFailURL), anyString(), any(SecureString.class), + any(CheckedSupplier.class), any(CheckedConsumer.class)); try { execute(randomBoolean() ? "auto" : "interactive", pathHomeParameter, "-b"); fail("Should have thrown exception"); @@ -277,11 +279,11 @@ public class SetupPasswordToolTests extends CommandTestCase { throw new RuntimeException("Did not properly parse password."); } - private URL checkURL(URL url) throws MalformedURLException { - return new URL(url, (url.getPath() + "/_xpack/security/_authenticate").replaceAll("/+", "/") + "?pretty"); + private URL checkURL(URL url) throws MalformedURLException, URISyntaxException { + return new URL(url, (url.toURI().getPath() + "/_xpack/security/_authenticate").replaceAll("/+", "/") + "?pretty"); } - private URL passwdURL(URL url, String user) throws MalformedURLException { - return new URL(url, (url.getPath() + "/_xpack/security/user/" + user + "/_password").replaceAll("/+", "/") + "?pretty"); + private URL passwdURL(URL url, String user) throws MalformedURLException, URISyntaxException { + return new URL(url, (url.toURI().getPath() + "/_xpack/security/user/" + user + "/_password").replaceAll("/+", "/") + "?pretty"); } } From 32d0c1b0c7356d2f60681fd9c16c899c45b6cfca Mon Sep 17 00:00:00 2001 From: lcawley Date: Fri, 17 Nov 2017 13:40:46 -0800 Subject: [PATCH 04/11] [DOCS] Re-enabled code snippet testing Original commit: elastic/x-pack-elasticsearch@31fd4c3668de90a14f9ede4b0990c0baefc34cd2 --- docs/en/rest-api/ml/put-datafeed.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/rest-api/ml/put-datafeed.asciidoc b/docs/en/rest-api/ml/put-datafeed.asciidoc index 4262afe46b9..a37b680e9f1 100644 --- a/docs/en/rest-api/ml/put-datafeed.asciidoc +++ b/docs/en/rest-api/ml/put-datafeed.asciidoc @@ -99,7 +99,7 @@ PUT _xpack/ml/datafeeds/datafeed-total-requests } -------------------------------------------------- // CONSOLE -// TEST[skip:https://github.com/elastic/elasticsearch/pull/27183] +// TEST[setup:server_metrics_job] When the {dfeed} is created, you receive the following results: [source,js] From 7c9af72d6e85a0ec5c2e259a8138e90ae169a8ab Mon Sep 17 00:00:00 2001 From: Nhat Nguyen Date: Fri, 17 Nov 2017 19:56:31 -0500 Subject: [PATCH 05/11] Removes the old handler for shard snapshot status Relates https://github.com/elastic/elasticsearch/pull/27443 Original commit: elastic/x-pack-elasticsearch@7f0021e3f45c969d3ac5593894edcb174be11e82 --- plugin/src/test/resources/org/elasticsearch/transport/handlers | 1 - 1 file changed, 1 deletion(-) diff --git a/plugin/src/test/resources/org/elasticsearch/transport/handlers b/plugin/src/test/resources/org/elasticsearch/transport/handlers index 4fa021261d3..b137379cb04 100644 --- a/plugin/src/test/resources/org/elasticsearch/transport/handlers +++ b/plugin/src/test/resources/org/elasticsearch/transport/handlers @@ -95,7 +95,6 @@ internal:cluster/nodes/indices/shard/store internal:cluster/nodes/indices/shard/store[n] internal:cluster/shard/failure internal:cluster/shard/started -internal:cluster/snapshot/update_snapshot internal:cluster/snapshot/update_snapshot_status internal:discovery/zen/fd/master_ping internal:discovery/zen/fd/ping From 316da9a9704b76f62663866c65f9f504df5e5ab5 Mon Sep 17 00:00:00 2001 From: Michael Basnight Date: Sat, 18 Nov 2017 21:43:25 -0600 Subject: [PATCH 06/11] Move the CLI into its own subproject (elastic/x-pack-elasticsearch#3032) relates elastic/elasticsearch#27114 Original commit: elastic/x-pack-elasticsearch@70e848822391275370a2a6f98315cc7d463bb428 --- plugin/build.gradle | 3 +++ 1 file changed, 3 insertions(+) diff --git a/plugin/build.gradle b/plugin/build.gradle index 877d748d3b3..f99cb4d5d25 100644 --- a/plugin/build.gradle +++ b/plugin/build.gradle @@ -56,6 +56,9 @@ configurations { } dependencies { + // CLI deps + compile project(path: ':core:cli', configuration: 'runtime') + // Request and Response objects compile "org.elasticsearch:x-pack-client-api-objects:${version}" From f8cb1e603b4c2420f95191e7525930caa3920f45 Mon Sep 17 00:00:00 2001 From: Michael Basnight Date: Sun, 19 Nov 2017 22:33:27 -0600 Subject: [PATCH 07/11] Removed unused Terminal param (elastic/x-pack-elasticsearch#2854) Relates elastic/elasticsearch#27216 Original commit: elastic/x-pack-elasticsearch@70bdda51ef415a453c2c0c6289974107bb59eb28 --- .../extensions/ListXPackExtensionCommandTests.java | 2 +- .../authc/esnative/ESNativeRealmMigrateToolTests.java | 2 +- .../authc/esnative/tool/SetupPasswordToolTests.java | 4 ++-- .../xpack/security/authc/file/tool/UsersToolTests.java | 10 +++++----- .../xpack/security/crypto/tool/SystemKeyToolTests.java | 2 +- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/plugin/src/test/java/org/elasticsearch/xpack/extensions/ListXPackExtensionCommandTests.java b/plugin/src/test/java/org/elasticsearch/xpack/extensions/ListXPackExtensionCommandTests.java index a4da235e2b0..5971d7604a9 100644 --- a/plugin/src/test/java/org/elasticsearch/xpack/extensions/ListXPackExtensionCommandTests.java +++ b/plugin/src/test/java/org/elasticsearch/xpack/extensions/ListXPackExtensionCommandTests.java @@ -51,7 +51,7 @@ public class ListXPackExtensionCommandTests extends ESTestCase { } @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { return env; } diff --git a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/ESNativeRealmMigrateToolTests.java b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/ESNativeRealmMigrateToolTests.java index 5d9338ae98a..a175a0f9b70 100644 --- a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/ESNativeRealmMigrateToolTests.java +++ b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/ESNativeRealmMigrateToolTests.java @@ -46,7 +46,7 @@ public class ESNativeRealmMigrateToolTests extends CommandTestCase { return new MigrateUserOrRoles() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { Settings.Builder builder = Settings.builder(); settings.forEach((k,v) -> builder.put(k, v)); return TestEnvironment.newEnvironment(builder.build()); diff --git a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java index 5327dc381f0..a0adb092dc8 100644 --- a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java +++ b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/esnative/tool/SetupPasswordToolTests.java @@ -102,7 +102,7 @@ public class SetupPasswordToolTests extends CommandTestCase { protected AutoSetup newAutoSetup() { return new AutoSetup() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { Settings.Builder builder = Settings.builder(); settings.forEach((k,v) -> builder.put(k, v)); return TestEnvironment.newEnvironment(builder.build()); @@ -114,7 +114,7 @@ public class SetupPasswordToolTests extends CommandTestCase { protected InteractiveSetup newInteractiveSetup() { return new InteractiveSetup() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { Settings.Builder builder = Settings.builder(); settings.forEach((k,v) -> builder.put(k, v)); return TestEnvironment.newEnvironment(builder.build()); diff --git a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/file/tool/UsersToolTests.java b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/file/tool/UsersToolTests.java index e76ab6f940f..2f3f52337b1 100644 --- a/plugin/src/test/java/org/elasticsearch/xpack/security/authc/file/tool/UsersToolTests.java +++ b/plugin/src/test/java/org/elasticsearch/xpack/security/authc/file/tool/UsersToolTests.java @@ -110,7 +110,7 @@ public class UsersToolTests extends CommandTestCase { protected AddUserCommand newAddUserCommand() { return new AddUserCommand() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { return new Environment(UsersToolTests.this.settings, confDir.getParent()); } }; @@ -120,7 +120,7 @@ public class UsersToolTests extends CommandTestCase { protected DeleteUserCommand newDeleteUserCommand() { return new DeleteUserCommand() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { return new Environment(UsersToolTests.this.settings, confDir.getParent()); } }; @@ -130,7 +130,7 @@ public class UsersToolTests extends CommandTestCase { protected PasswordCommand newPasswordCommand() { return new PasswordCommand() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { return new Environment(UsersToolTests.this.settings, confDir.getParent()); } }; @@ -140,7 +140,7 @@ public class UsersToolTests extends CommandTestCase { protected RolesCommand newRolesCommand() { return new RolesCommand() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { return new Environment(UsersToolTests.this.settings, confDir.getParent()); } }; @@ -150,7 +150,7 @@ public class UsersToolTests extends CommandTestCase { protected ListCommand newListCommand() { return new ListCommand() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { return new Environment(UsersToolTests.this.settings, confDir.getParent()); } }; diff --git a/plugin/src/test/java/org/elasticsearch/xpack/security/crypto/tool/SystemKeyToolTests.java b/plugin/src/test/java/org/elasticsearch/xpack/security/crypto/tool/SystemKeyToolTests.java index 2e47f6f3100..e0694ee53c9 100644 --- a/plugin/src/test/java/org/elasticsearch/xpack/security/crypto/tool/SystemKeyToolTests.java +++ b/plugin/src/test/java/org/elasticsearch/xpack/security/crypto/tool/SystemKeyToolTests.java @@ -49,7 +49,7 @@ public class SystemKeyToolTests extends CommandTestCase { return new SystemKeyTool() { @Override - protected Environment createEnv(Terminal terminal, Map settings) throws UserException { + protected Environment createEnv(Map settings) throws UserException { Settings.Builder builder = Settings.builder(); settings.forEach((k,v) -> builder.put(k, v)); return TestEnvironment.newEnvironment(builder.build()); From 040be7da9d29b9c40405efb7655f3b632ffc2d9b Mon Sep 17 00:00:00 2001 From: David Kyle Date: Mon, 20 Nov 2017 13:32:14 +0000 Subject: [PATCH 08/11] =?UTF-8?q?[DOCS][TESTS]=20Don=E2=80=99t=20randomise?= =?UTF-8?q?=20content=20type=20for=20docs=20tests.=20(elastic/x-pack-elast?= =?UTF-8?q?icsearch#3058)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Original commit: elastic/x-pack-elasticsearch@7617901ff805d5534e4c54964bdd1c33f6457623 --- .../elasticsearch/smoketest/XDocsClientYamlTestSuiteIT.java | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/src/test/java/org/elasticsearch/smoketest/XDocsClientYamlTestSuiteIT.java b/docs/src/test/java/org/elasticsearch/smoketest/XDocsClientYamlTestSuiteIT.java index 0adff1f9e27..b208d36372a 100644 --- a/docs/src/test/java/org/elasticsearch/smoketest/XDocsClientYamlTestSuiteIT.java +++ b/docs/src/test/java/org/elasticsearch/smoketest/XDocsClientYamlTestSuiteIT.java @@ -88,7 +88,6 @@ public class XDocsClientYamlTestSuiteIT extends ESClientYamlSuiteTestCase { if (state.equals("started") == false || state.equals("starting") == false) { getAdminExecutionContext().callApi("xpack.watcher.start", emptyMap(), emptyList(), emptyMap()); } - assertThat(state, is("started")); }); } @@ -122,4 +121,9 @@ public class XDocsClientYamlTestSuiteIT extends ESClientYamlSuiteTestCase { public void cleanMlState() { new MlRestTestStateCleaner(logger, adminClient(), this); } + + @Override + protected boolean randomizeContentType() { + return false; + } } From 5ec688a9311b40e0b685dbd104687ce959f0b326 Mon Sep 17 00:00:00 2001 From: lcawley Date: Mon, 20 Nov 2017 07:22:59 -0800 Subject: [PATCH 09/11] [DOCS] Removed beta warning from ML tutorial Original commit: elastic/x-pack-elasticsearch@38d480fe060d73d39bc17fad21b7a1dc106cf982 --- docs/en/ml/getting-started.asciidoc | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/en/ml/getting-started.asciidoc b/docs/en/ml/getting-started.asciidoc index d582381da22..4ca634adb07 100644 --- a/docs/en/ml/getting-started.asciidoc +++ b/docs/en/ml/getting-started.asciidoc @@ -34,17 +34,13 @@ To follow the steps in this tutorial, you will need the following components of the Elastic Stack: * {es} {version}, which stores the data and the analysis results -* {xpack} {version}, which includes the beta {ml} features for both {es} and {kib} +* {xpack} {version}, which includes the {ml} features for both {es} and {kib} * {kib} {version}, which provides a helpful user interface for creating and viewing jobs + //ll {ml} features are available to use as an API, however this tutorial //will focus on using the {ml} tab in the {kib} UI. -WARNING: The {xpackml} features are in beta and subject to change. -Beta features are not subject to the same support SLA as GA features, -and deployment in production is at your own risk. - See the https://www.elastic.co/support/matrix[Elastic Support Matrix] for information about supported operating systems. From 712e740761d88a3612adef01f9b620e4f58b8996 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Mon, 20 Nov 2017 08:44:43 -0800 Subject: [PATCH 10/11] [DOCS] Added sysgenkey command and watcher encryption settings (elastic/x-pack-elasticsearch#3043) * [DOCS] Added sysgenkey command and watcher settings * [DOCS] Added data encryption task for Watcher * [DOCS] Addressed feedback about watcher encryption Original commit: elastic/x-pack-elasticsearch@edb1fccbfbaff6bd3e32f0de1195f42e9bcea653 --- docs/en/commands/syskeygen.asciidoc | 34 +++++------- .../settings/notification-settings.asciidoc | 12 ++--- docs/en/watcher/encrypting-data.asciidoc | 53 +++++++++++++++++++ docs/en/watcher/index.asciidoc | 2 + 4 files changed, 72 insertions(+), 29 deletions(-) create mode 100644 docs/en/watcher/encrypting-data.asciidoc diff --git a/docs/en/commands/syskeygen.asciidoc b/docs/en/commands/syskeygen.asciidoc index be8708ecd37..fd4c9235f74 100644 --- a/docs/en/commands/syskeygen.asciidoc +++ b/docs/en/commands/syskeygen.asciidoc @@ -2,15 +2,7 @@ [[syskeygen]] == syskeygen -The `syskeygen` command creates a system key file in -`CONFIG_DIR/x-pack/system_key`. You can then copy the generated system key to -the rest of the nodes in the cluster. - -//TBD: Why? What's its purpose? - -IMPORTANT: The system key is a symmetric key, so the same key must be on every -node in the cluster. You must also ensure that the file permissions are set -such that the key can be read by the user that {es} runs as. +The `syskeygen` command creates a system key file in `CONFIG_DIR/x-pack`. [float] === Synopsis @@ -25,21 +17,19 @@ bin/x-pack/syskeygen [float] === Description -Encryption of sensitive data requires a key. For example, if you enable the -<>, -//TBD: Can we clarify what type of sensitive data is encrypted when this is enabled? -you must also create a system key file and place it in the secure setting store. -After you run the `syskeygen` command, run the -`elasticsearch-keystore add-file xpack.watcher.encryption_key ` -command. Finally, remove the `system_key` file from the filesystem. Repeat these -steps on every node in the cluster. +The command generates a `system_key` file, which you can use to symmetrically +encrypt sensitive data. For example, you can use this key to prevent {watcher} +from returning and storing information that contains clear text credentials. See {xpack-ref}/encrypting-data.html[Encrypting sensitive data in {watcher}]. -//TBD: Are there other scenarios where you could use this system file? +IMPORTANT: The system key is a symmetric key, so the same key must be used on +every node in the cluster. [float] === Parameters -`-E `:: Configures a setting. +`-E `:: Configures a setting. For example, if you have a custom +installation of {es}, you can use this parameter to specify the `ES_PATH_CONF` +environment variable. `-h, --help`:: Returns all of the command parameters. @@ -47,14 +37,14 @@ steps on every node in the cluster. `-v, --verbose`:: Shows verbose output. -//// + [float] === Examples -//TBD: What's an example of a -E value that we might use in this command? +The following command generates a `system_key` file in the +default `$ES_HOME/config/x-pack` directory: [source, sh] -------------------------------------------------- bin/x-pack/syskeygen -------------------------------------------------- -//// diff --git a/docs/en/settings/notification-settings.asciidoc b/docs/en/settings/notification-settings.asciidoc index 65d536e7e7c..2784aa88b80 100644 --- a/docs/en/settings/notification-settings.asciidoc +++ b/docs/en/settings/notification-settings.asciidoc @@ -24,17 +24,15 @@ Set to `false` to disable {watcher} on the node. `xpack.watcher.encrypt_sensitive_data` (<>):: Set to `true` to encrypt sensitive data. If this setting is enabled, you -must also specify the `xpack.watcher.encryption_key` setting. -//TBD: Can we be more specific about what type of watcher data it's encrypting? +must also specify the `xpack.watcher.encryption_key` setting. For more +information, see +{xpack-ref}/encrypting-data.html[Encrypting sensitive data in {watcher}]. `xpack.watcher.encryption_key` (<>):: Specifies the path to a file that contains a key for encrypting sensitive data. If `xpack.watcher.encrypt_sensitive_data` is set to `true`, this setting is -required. For example, to generate the key file, run the <> tool. -Then import the key into the secure setting store with the -`elasticsearch-keystore add-file xpack.watcher.encryption_key ` -command. Finally, remove the system_key file from the filesystem. Repeat this on -every node in the cluster. +required. For more information, see +{xpack-ref}/encrypting-data.html[Encrypting sensitive data in {watcher}]. `xpack.watcher.history.cleaner_service.enabled`:: Set to `false` (default) to disable the cleaner service, which removes previous diff --git a/docs/en/watcher/encrypting-data.asciidoc b/docs/en/watcher/encrypting-data.asciidoc new file mode 100644 index 00000000000..5f2a55ac6a7 --- /dev/null +++ b/docs/en/watcher/encrypting-data.asciidoc @@ -0,0 +1,53 @@ +[[encrypting-data]] +== Encrypting Sensitive Data in {watcher} + +Watches might have access to sensitive data such as HTTP basic authentication +information or details about your SMTP email service. You can encrypt this +data by generating a key and adding some secure settings on each node in your +cluster. + +To encrypt sensitive data in {watcher}: + +. Use the {ref}/syskeygen.html[syskeygen] command to create a system key file. + +. Copy the `system_key` file to all of the nodes in your cluster. ++ +-- +IMPORTANT: The system key is a symmetric key, so the same key must be used on +every node in the cluster. + +-- + +. Set the +{ref}/notification-settings.html[`xpack.watcher.encrypt_sensitive_data` setting] +in the {ref}/secure-settings.html[{es} keystore] on each node in the cluster. ++ +-- + +For example, run the following commands to create and update the keystore: + +[source,sh] +---------------------------------------------------------------- +bin/elasticsearch-keystore create +bin/elasticsearch-keystore add xpack.watcher.encrypt_sensitive_data true +---------------------------------------------------------------- +-- + +. Set the +{ref}/notification-settings.html[`xpack.watcher.encryption_key` setting] in the +{ref}/secure-settings.html[{es} keystore] on each node in the cluster. ++ +-- +For example, run the following command to import the `system_key` file on +each node: + +[source,sh] +---------------------------------------------------------------- +bin/elasticsearch-keystore add-file xpack.watcher.encryption_key /system_key +---------------------------------------------------------------- +-- + +. Delete the `system_key` file on each node in the cluster. + +NOTE: Existing watches are not affected by these changes. Only watches that you +create after following these steps have encryption enabled. diff --git a/docs/en/watcher/index.asciidoc b/docs/en/watcher/index.asciidoc index 65e5be0d468..0fa92675f4b 100644 --- a/docs/en/watcher/index.asciidoc +++ b/docs/en/watcher/index.asciidoc @@ -74,6 +74,8 @@ include::getting-started.asciidoc[] include::how-watcher-works.asciidoc[] +include::encrypting-data.asciidoc[] + include::input.asciidoc[] include::trigger.asciidoc[] From 77803b06c501f875666db22f64e97511e3c4dc33 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Mon, 20 Nov 2017 11:27:10 -0800 Subject: [PATCH 11/11] [DOCS] Remove http.ssl.enabled step from installation (elastic/x-pack-elasticsearch#3065) Original commit: elastic/x-pack-elasticsearch@b071fa191f4e62adf373edc1d45f0638f06573b1 --- docs/en/setup/installing-xes.asciidoc | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/en/setup/installing-xes.asciidoc b/docs/en/setup/installing-xes.asciidoc index b397ce29bf0..dc7b80a85c4 100644 --- a/docs/en/setup/installing-xes.asciidoc +++ b/docs/en/setup/installing-xes.asciidoc @@ -189,15 +189,12 @@ xpack.ssl.key: certs/${node.name}/${node.name}.key <1> xpack.ssl.certificate: certs/${node.name}/${node.name}.crt <2> xpack.ssl.certificate_authorities: certs/ca/ca.crt <3> xpack.security.transport.ssl.enabled: true -xpack.security.http.ssl.enabled: true <4> ----------------------------------------------------------- <1> If this path does not exist on every node or the file name does not match the `node.name` configuration setting, you must specify the full path to the node key file. <2> Alternatively, specify the full path to the node certificate. <3> Alternatively specify the full path to the CA certificate. -<4> This setting is optional. It enables SSL on the HTTP layer to ensure that -communication between HTTP clients and the cluster is encrypted. -- .. Start {es}.