From 10827033c5beac283ef0ca7367c93563c4176652 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Mon, 22 Jan 2018 13:52:47 -0800 Subject: [PATCH 01/13] [DOCS] Added information about overall bucket scores (elastic/x-pack-elasticsearch#3333) Original commit: elastic/x-pack-elasticsearch@68efc63f25f8688f5099359114eabe34a04d3b4d --- docs/en/ml/buckets.asciidoc | 32 +++++++++++++++++++++++--------- docs/en/ml/overview.asciidoc | 2 +- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/en/ml/buckets.asciidoc b/docs/en/ml/buckets.asciidoc index fb69abe0cec..89d7ea8cdea 100644 --- a/docs/en/ml/buckets.asciidoc +++ b/docs/en/ml/buckets.asciidoc @@ -1,12 +1,26 @@ -[float] [[ml-buckets]] === Buckets +++++ +Buckets +++++ -The {xpackml} features use the concept of a bucket to divide the time -series into batches for processing. The _bucket span_ is part of the -configuration information for a job. It defines the time interval that is used -to summarize and model the data. This is typically between 5 minutes to 1 hour -and it depends on your data characteristics. When you set the bucket span, -take into account the granularity at which you want to analyze, the frequency -of the input data, the typical duration of the anomalies, and the frequency at -which alerting is required. +The {xpackml} features use the concept of a _bucket_ to divide the time series +into batches for processing. + +The _bucket span_ is part of the configuration information for a job. It defines +the time interval that is used to summarize and model the data. This is +typically between 5 minutes to 1 hour and it depends on your data characteristics. +When you set the bucket span, take into account the granularity at which you +want to analyze, the frequency of the input data, the typical duration of the +anomalies, and the frequency at which alerting is required. + +When you view your {ml} results, each bucket has an anomaly score. This score is +a statistically aggregated and normalized view of the combined anomalousness of +all the record results in the bucket. If you have more than one job, you can +also obtain overall bucket results, which combine and correlate anomalies from +multiple jobs into an overall score. When you view the results for jobs groups +in {kib}, it provides the overall bucket scores. + +For more information, see +{ref}/ml-results-resource.html[Results Resources] and +{ref}/ml-get-overall-buckets.html[Get Overall Buckets API]. diff --git a/docs/en/ml/overview.asciidoc b/docs/en/ml/overview.asciidoc index 7c58774188b..a13fb58f1f6 100644 --- a/docs/en/ml/overview.asciidoc +++ b/docs/en/ml/overview.asciidoc @@ -3,6 +3,7 @@ include::analyzing.asciidoc[] include::forecasting.asciidoc[] +include::buckets.asciidoc[] include::calendars.asciidoc[] [[ml-concepts]] @@ -16,5 +17,4 @@ concepts from the outset will tremendously help ease the learning process. include::jobs.asciidoc[] include::datafeeds.asciidoc[] -include::buckets.asciidoc[] include::architecture.asciidoc[] From c0edf2197bbc04edcf139017a44d259963192999 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Mon, 22 Jan 2018 15:15:31 -0800 Subject: [PATCH 02/13] [DOCS] Replaced settings with links (elastic/x-pack-elasticsearch#3626) Original commit: elastic/x-pack-elasticsearch@4ad018521e19adc5b30566d18599b02a82234754 --- docs/en/security/auditing.asciidoc | 63 ++++++------------------------ 1 file changed, 13 insertions(+), 50 deletions(-) diff --git a/docs/en/security/auditing.asciidoc b/docs/en/security/auditing.asciidoc index 2fd6a1c3548..8bff8727f83 100644 --- a/docs/en/security/auditing.asciidoc +++ b/docs/en/security/auditing.asciidoc @@ -304,7 +304,7 @@ The format of a log entry is: `` :: Information about the local node that generated the log entry. You can control what node information is included by configuring the - <>. + {ref}/auditing-settings.html#node-audit-settings[local node info settings]. `` :: The layer from which this event originated: `rest`, `transport` or `ip_filter`. `` :: The type of event that occurred: `anonymous_access_denied`, @@ -321,35 +321,13 @@ The format of a log entry is: === Logfile Output Settings The events and some other information about what gets logged can be -controlled using settings in the `elasticsearch.yml` file. - -.Audited Event Settings -[cols="4,^2,4",options="header"] -|====== -| Name | Default | Description -| `xpack.security.audit.logfile.events.include` | `access_denied`, `access_granted`, `anonymous_access_denied`, `authentication_failed`, `connection_denied`, `tampered_request`, `run_as_denied`, `run_as_granted` | Includes the specified events in the output. -| `xpack.security.audit.logfile.events.exclude` | | Excludes the specified events from the output. -| `xpack.security.audit.logfile.events.emit_request_body`| false | Include or exclude the request body from REST requests - on certain event types such as `authentication_failed`. -|====== - +controlled using settings in the `elasticsearch.yml` file. See +{ref}/auditing-settings.html#event-audit-settings[Audited Event Settings] and +{ref}/auditing-settings.html#node-audit-settings[Local Node Info Settings]. IMPORTANT: No filtering is performed when auditing, so sensitive data may be audited in plain text when including the request body in audit events. -[[audit-log-entry-local-node-info]] -.Local Node Info Settings -[cols="4,^2,4",options="header"] -|====== -| Name | Default | Description -| `xpack.security.audit.logfile.prefix.emit_node_name` | true | Include or exclude the node's name - from the local node info. -| `xpack.security.audit.logfile.prefix.emit_node_host_address` | false | Include or exclude the node's IP address - from the local node info. -| `xpack.security.audit.logfile.prefix.emit_node_host_name` | false | Include or exclude the node's host name - from the local node info. -|====== - [[logging-file]] You can also configure how the logfile is written in the `log4j2.properties` file located in `CONFIG_DIR/x-pack`. By default, audit information is appended to the @@ -450,19 +428,8 @@ in the `elasticsearch.yml` file: xpack.security.audit.outputs: [ index, logfile ] ---------------------------- -.Audit Log Indexing Configuration -[options="header"] -|====== -| Attribute | Default Setting | Description -| `xpack.security.audit.index.bulk_size` | `1000` | Controls how many audit events are batched into a single write. -| `xpack.security.audit.index.flush_interval` | `1s` | Controls how often buffered events are flushed to the index. -| `xpack.security.audit.index.rollover` | `daily` | Controls how often to roll over to a new index: - `hourly`, `daily`, `weekly`, or `monthly`. -| `xpack.security.audit.index.events.include` | `anonymous_access_denied`, `authentication_failed`, `realm_authentication_failed`, `access_granted`, `access_denied`, `tampered_request`, `connection_granted`, `connection_denied`, `run_as_granted`, `run_as_denied` | The audit events to be indexed. See <> for the complete list. -| `xpack.security.audit.index.events.exclude` | | The audit events to exclude from indexing. -| `xpack.security.audit.index.events.emit_request_body`| false | Include or exclude the request body from REST requests - on certain event types such as `authentication_failed`. -|====== +For more configuration options, see +{ref}/auditing-settings.html#index-audit-settings[Audit Log Indexing Configuration Settings]. IMPORTANT: No filtering is performed when auditing, so sensitive data may be audited in plain text when including the request body in audit events. @@ -487,18 +454,14 @@ xpack.security.audit.index.settings: ==== Forwarding Audit Logs to a Remote Cluster To index audit events to a remote Elasticsearch cluster, you configure -the following `xpack.security.audit.index.client` settings. +the following `xpack.security.audit.index.client` settings: -.Remote Audit Log Indexing Configuration -[options="header"] -|====== -| Attribute | Description -| `xpack.security.audit.index.client.hosts` | Comma-separated list of `host:port` pairs. These hosts - should be nodes in the remote cluster. -| `xpack.security.audit.index.client.cluster.name` | The name of the remote cluster. -| `xpack.security.audit.index.client.xpack.security.user` | The `username:password` pair to use to authenticate with - the remote cluster. -|====== +* `xpack.security.audit.index.client.hosts` +* `xpack.security.audit.index.client.cluster.name` +* `xpack.security.audit.index.client.xpack.security.user` + +For more information about these settings, see +{ref}/auditing-settings.html#remote-audit-settings[Remote Audit Log Indexing Configuration Settings]. You can pass additional settings to the remote client by specifying them in the `xpack.security.audit.index.client` namespace. For example, to allow the remote From 9d87b63ca4dc8eaebbdb355fe0672d3c6f2b6e25 Mon Sep 17 00:00:00 2001 From: Ryan Ernst Date: Mon, 22 Jan 2018 22:58:34 -0800 Subject: [PATCH 03/13] Build: Fix third party audit task for xpack core (elastic/x-pack-elasticsearch#3656) This commit re-enables thirdPartyAudit for x-pack core. Previously, when xpack was a single plugin, it transitively picked up httpcore-nio through the elasticsearch rest client. Now that xpack core does not depend on the rest client, httpcore-nio must be added as a dependency. Additionally, commons-logging was previously handled through the rest client, but now xpack depends directly on this, thus excludes must be added for the pesky missing classes there. This commit also cleans up unnecessary parts of plugin/build.gradle no longer necessary. Original commit: elastic/x-pack-elasticsearch@70e936bdc3b14026729c725ac1b9478a0b81017e --- plugin/build.gradle | 19 +- plugin/core/build.gradle | 174 ++---------------- .../licenses/httpcore-nio-4.4.5.jar.sha1 | 0 3 files changed, 26 insertions(+), 167 deletions(-) rename plugin/{security => core}/licenses/httpcore-nio-4.4.5.jar.sha1 (100%) diff --git a/plugin/build.gradle b/plugin/build.gradle index 6f5ebc75ae4..843601cc4c9 100644 --- a/plugin/build.gradle +++ b/plugin/build.gradle @@ -19,10 +19,6 @@ es_meta_plugin { 'ml', 'monitoring', 'security', 'upgrade', 'watcher'] } -ext.expansions = [ - 'project.version': version, -] - dependencies { testCompile project(path: ':x-pack-elasticsearch:plugin:core', configuration: 'testArtifacts') } @@ -40,6 +36,13 @@ artifacts { } integTestRunner { + /* + * We have to disable setting the number of available processors as tests in the same JVM randomize processors and will step on each + * other if we allow them to set the number of available processors as it's set-once in Netty. + */ + systemProperty 'es.set.netty.runtime.available.processors', 'false' + + // TODO: fix this rest test to not depend on a hardcoded port! def blacklist = ['getting_started/10_monitor_cluster_health/*'] boolean snapshot = "true".equals(System.getProperty("build.snapshot", "true")) @@ -140,14 +143,6 @@ integTestCluster { } } -integTestRunner { - /* - * We have to disable setting the number of available processors as tests in the same JVM randomize processors and will step on each - * other if we allow them to set the number of available processors as it's set-once in Netty. - */ - systemProperty 'es.set.netty.runtime.available.processors', 'false' -} - run { setting 'xpack.ml.enabled', 'true' setting 'xpack.graph.enabled', 'true' diff --git a/plugin/core/build.gradle b/plugin/core/build.gradle index 5e14e72534f..6419bc3c965 100644 --- a/plugin/core/build.gradle +++ b/plugin/core/build.gradle @@ -5,6 +5,9 @@ import java.nio.file.Path import java.nio.file.StandardCopyOption apply plugin: 'elasticsearch.esplugin' + +archivesBaseName = 'x-pack-core' + esplugin { name 'x-pack-core' description 'Elasticsearch Expanded Pack Plugin - Core' @@ -18,27 +21,16 @@ esplugin { integTest.enabled = false dependencyLicenses { - mapping from: /netty-.*/, to: 'netty' mapping from: /bc.*/, to: 'bouncycastle' - mapping from: /owasp-java-html-sanitizer.*/, to: 'owasp-java-html-sanitizer' - mapping from: /transport-netty.*/, to: 'elasticsearch' - mapping from: /transport-nio.*/, to: 'elasticsearch' - mapping from: /elasticsearch-nio.*/, to: 'elasticsearch' - mapping from: /elasticsearch-rest-client.*/, to: 'elasticsearch' mapping from: /http.*/, to: 'httpclient' // pulled in by rest client mapping from: /commons-.*/, to: 'commons' // pulled in by rest client - ignoreSha 'elasticsearch-rest-client' - ignoreSha 'transport-netty4' - ignoreSha 'transport-nio' - ignoreSha 'elasticsearch-nio' - ignoreSha 'elasticsearch-rest-client-sniffer' - ignoreSha 'x-pack-core' } dependencies { provided "org.elasticsearch:elasticsearch:${version}" compile "org.apache.httpcomponents:httpclient:${versions.httpclient}" compile "org.apache.httpcomponents:httpcore:${versions.httpcore}" + compile "org.apache.httpcomponents:httpcore-nio:${versions.httpcore}" compile "org.apache.httpcomponents:httpasyncclient:${versions.httpasyncclient}" compile "commons-logging:commons-logging:${versions.commonslogging}" @@ -50,7 +42,6 @@ dependencies { compile 'org.bouncycastle:bcpkix-jdk15on:1.58' compile project(path: ':modules:transport-netty4', configuration: 'runtime') - //testCompile project(path: ':core:cli', configuration: 'runtime') testCompile 'org.elasticsearch:securemock:1.2' testCompile "org.elasticsearch:mocksocket:${versions.mocksocket}" testCompile "org.apache.logging.log4j:log4j-slf4j-impl:${versions.log4j}" @@ -60,6 +51,10 @@ dependencies { testCompile project(path: ':modules:analysis-common', configuration: 'runtime') } +ext.expansions = [ + 'project.version': version +] + processResources { from(sourceSets.main.resources.srcDirs) { exclude '**/public.key' @@ -81,14 +76,9 @@ forbiddenPatterns { exclude '**/*.zip' } -archivesBaseName = 'x-pack-core' - compileJava.options.compilerArgs << "-Xlint:-deprecation,-rawtypes,-serial,-try,-unchecked" compileTestJava.options.compilerArgs << "-Xlint:-deprecation,-rawtypes,-serial,-try,-unchecked" -// TODO: fix these! -thirdPartyAudit.enabled = false - licenseHeaders { approvedLicenses << 'BCrypt (BSD-like)' additionalLicense 'BCRYP', 'BCrypt (BSD-like)', 'Copyright (c) 2006 Damien Miller ' @@ -100,6 +90,7 @@ sourceSets.test.java { srcDir '../../license-tools/src/main/java' } +// TODO: remove this jar once xpack extensions have been removed // assemble the API JAR for the transport-client and extension authors; this JAR is the core JAR by another name project.afterEvaluate { task apiJar { @@ -131,108 +122,6 @@ project.afterEvaluate { } } -// -// integTestRunner { -// // TODO: fix this rest test to not depend on a hardcoded port! -// def blacklist = ['getting_started/10_monitor_cluster_health/*'] -// boolean snapshot = "true".equals(System.getProperty("build.snapshot", "true")) -// if (!snapshot) { -// // these tests attempt to install basic/internal licenses signed against the dev/public.key -// // Since there is no infrastructure in place (anytime soon) to generate licenses using the production -// // private key, these tests are whitelisted in non-snapshot test runs -// blacklist.addAll(['xpack/15_basic/*', 'license/20_put_license/*']) -// } -// systemProperty 'tests.rest.blacklist', blacklist.join(',') -// } - -// // location of generated keystores and certificates -// File keystoreDir = new File(project.buildDir, 'keystore') - -// // Generate the node's keystore -// File nodeKeystore = new File(keystoreDir, 'test-node.jks') -// task createNodeKeyStore(type: LoggedExec) { -// doFirst { -// if (nodeKeystore.parentFile.exists() == false) { -// nodeKeystore.parentFile.mkdirs() -// } -// if (nodeKeystore.exists()) { -// delete nodeKeystore -// } -// } -// executable = new File(project.javaHome, 'bin/keytool') -// standardInput = new ByteArrayInputStream('FirstName LastName\nUnit\nOrganization\nCity\nState\nNL\nyes\n\n'.getBytes('UTF-8')) -// args '-genkey', -// '-alias', 'test-node', -// '-keystore', nodeKeystore, -// '-keyalg', 'RSA', -// '-keysize', '2048', -// '-validity', '712', -// '-dname', 'CN=smoke-test-plugins-ssl', -// '-keypass', 'keypass', -// '-storepass', 'keypass' -// } - -// Add keystores to test classpath: it expects it there -//sourceSets.test.resources.srcDir(keystoreDir) -//processTestResources.dependsOn(createNodeKeyStore) - -// integTestCluster { -// dependsOn createNodeKeyStore -// setting 'xpack.ml.enabled', 'true' -// setting 'logger.org.elasticsearch.xpack.ml.datafeed', 'TRACE' -// // Integration tests are supposed to enable/disable exporters before/after each test -// setting 'xpack.monitoring.exporters._local.type', 'local' -// setting 'xpack.monitoring.exporters._local.enabled', 'false' -// setting 'xpack.monitoring.collection.interval', '-1' -// setting 'xpack.security.authc.token.enabled', 'true' -// setting 'xpack.security.transport.ssl.enabled', 'true' -// setting 'xpack.security.transport.ssl.keystore.path', nodeKeystore.name -// setting 'xpack.security.transport.ssl.verification_mode', 'certificate' -// setting 'xpack.security.audit.enabled', 'true' -// keystoreSetting 'bootstrap.password', 'x-pack-test-password' -// keystoreSetting 'xpack.security.transport.ssl.keystore.secure_password', 'keypass' -// distribution = 'zip' // this is important since we use the reindex module in ML - -// setupCommand 'setupTestUser', 'bin/x-pack/users', 'useradd', 'x_pack_rest_user', '-p', 'x-pack-test-password', '-r', 'superuser' - -// extraConfigFile nodeKeystore.name, nodeKeystore - -// waitCondition = { NodeInfo node, AntBuilder ant -> -// File tmpFile = new File(node.cwd, 'wait.success') - -// for (int i = 0; i < 10; i++) { -// // we use custom wait logic here as the elastic user is not available immediately and ant.get will fail when a 401 is returned -// HttpURLConnection httpURLConnection = null; -// try { -// httpURLConnection = (HttpURLConnection) new URL("http://${node.httpUri()}/_cluster/health?wait_for_nodes=${numNodes}&wait_for_status=yellow").openConnection(); -// httpURLConnection.setRequestProperty("Authorization", "Basic " + -// Base64.getEncoder().encodeToString("x_pack_rest_user:x-pack-test-password".getBytes(StandardCharsets.UTF_8))); -// httpURLConnection.setRequestMethod("GET"); -// httpURLConnection.connect(); -// if (httpURLConnection.getResponseCode() == 200) { -// tmpFile.withWriter StandardCharsets.UTF_8.name(), { -// it.write(httpURLConnection.getInputStream().getText(StandardCharsets.UTF_8.name())) -// } -// } -// } catch (Exception e) { -// if (i == 9) { -// logger.error("final attempt of calling cluster health failed", e) -// } else { -// logger.debug("failed to call cluster health", e) -// } -// } finally { -// if (httpURLConnection != null) { -// httpURLConnection.disconnect(); -// } -// } - -// // did not start, so wait a bit before trying again -// Thread.sleep(500L); -// } -// return tmpFile.exists() -// } -//} - test { /* * We have to disable setting the number of available processors as tests in the same JVM randomize processors and will step on each @@ -249,6 +138,7 @@ integTestRunner { systemProperty 'es.set.netty.runtime.available.processors', 'false' } + // TODO: don't publish test artifacts just to run messy tests, fix the tests! // https://github.com/elastic/x-plugins/issues/724 configurations { @@ -264,38 +154,12 @@ artifacts { testArtifacts testJar } -// pulled in as external dependency to work on java 9 -if (JavaVersion.current() <= JavaVersion.VERSION_1_8) { - thirdPartyAudit.excludes += [ - 'com.sun.activation.registries.MailcapParseException', - 'javax.activation.ActivationDataFlavor', - 'javax.activation.CommandInfo', - 'javax.activation.CommandMap', - 'javax.activation.CommandObject', - 'javax.activation.DataContentHandler', - 'javax.activation.DataContentHandlerFactory', - 'javax.activation.DataHandler$1', - 'javax.activation.DataHandler', - 'javax.activation.DataHandlerDataSource', - 'javax.activation.DataSource', - 'javax.activation.DataSourceDataContentHandler', - 'javax.activation.FileDataSource', - 'javax.activation.FileTypeMap', - 'javax.activation.MimeType', - 'javax.activation.MimeTypeParameterList', - 'javax.activation.MimeTypeParseException', - 'javax.activation.ObjectDataContentHandler', - 'javax.activation.SecuritySupport$1', - 'javax.activation.SecuritySupport$2', - 'javax.activation.SecuritySupport$3', - 'javax.activation.SecuritySupport$4', - 'javax.activation.SecuritySupport$5', - 'javax.activation.SecuritySupport', - 'javax.activation.URLDataSource', - 'javax.activation.UnsupportedDataTypeException' - ] -} - -run { - distribution = 'zip' -} +thirdPartyAudit.excludes = [ + //commons-logging optional dependencies + 'org.apache.avalon.framework.logger.Logger', + 'org.apache.log.Hierarchy', + 'org.apache.log.Logger', + //commons-logging provided dependencies + 'javax.servlet.ServletContextEvent', + 'javax.servlet.ServletContextListener' +] diff --git a/plugin/security/licenses/httpcore-nio-4.4.5.jar.sha1 b/plugin/core/licenses/httpcore-nio-4.4.5.jar.sha1 similarity index 100% rename from plugin/security/licenses/httpcore-nio-4.4.5.jar.sha1 rename to plugin/core/licenses/httpcore-nio-4.4.5.jar.sha1 From c3efa4b6bc27ab3eab7c5e10f5227275dc6a13e0 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 23 Jan 2018 10:28:37 +0100 Subject: [PATCH 04/13] [TEST] disable ML when ML is not installed Original commit: elastic/x-pack-elasticsearch@cd84acc3e05c381c81cfa72e71c8dbe4d990d83a --- .../monitoring/exporter/local/LocalExporterIntegTestCase.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugin/monitoring/src/test/java/org/elasticsearch/xpack/monitoring/exporter/local/LocalExporterIntegTestCase.java b/plugin/monitoring/src/test/java/org/elasticsearch/xpack/monitoring/exporter/local/LocalExporterIntegTestCase.java index ec9ad937794..f3f8e952d3a 100644 --- a/plugin/monitoring/src/test/java/org/elasticsearch/xpack/monitoring/exporter/local/LocalExporterIntegTestCase.java +++ b/plugin/monitoring/src/test/java/org/elasticsearch/xpack/monitoring/exporter/local/LocalExporterIntegTestCase.java @@ -51,7 +51,7 @@ public abstract class LocalExporterIntegTestCase extends MonitoringIntegTestCase .put("xpack.monitoring.exporters." + exporterName + ".type", LocalExporter.TYPE) .put("xpack.monitoring.exporters." + exporterName + ".enabled", false) .put("xpack.monitoring.exporters." + exporterName + "." + CLUSTER_ALERTS_MANAGEMENT_SETTING, false) -// .put(XPackSettings.WATCHER_ENABLED.getKey(), false) + .put(XPackSettings.MACHINE_LEARNING_ENABLED.getKey(), false) .put(NetworkModule.HTTP_ENABLED.getKey(), false) .build(); } From 8f393c30665f8bf05099010650e5033c2cd49fed Mon Sep 17 00:00:00 2001 From: David Kyle Date: Tue, 23 Jan 2018 09:34:07 +0000 Subject: [PATCH 05/13] [ML] Improve error message when creating calendars (elastic/x-pack-elasticsearch#3668) Original commit: elastic/x-pack-elasticsearch@996b0e2f650799024eb30742b7124511eafd1d54 --- .../xpack/ml/action/TransportPutCalendarAction.java | 10 ++++++++-- .../resources/rest-api-spec/test/ml/calendar_crud.yml | 2 +- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/action/TransportPutCalendarAction.java b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/action/TransportPutCalendarAction.java index bbec8233449..f8a3a5cecdc 100644 --- a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/action/TransportPutCalendarAction.java +++ b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/action/TransportPutCalendarAction.java @@ -22,6 +22,7 @@ import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.xcontent.ToXContent; import org.elasticsearch.common.xcontent.XContentBuilder; import org.elasticsearch.common.xcontent.XContentFactory; +import org.elasticsearch.index.engine.VersionConflictEngineException; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.transport.TransportService; import org.elasticsearch.xpack.ml.MLMetadataField; @@ -83,8 +84,13 @@ public class TransportPutCalendarAction extends HandledTransportAction Date: Tue, 23 Jan 2018 13:05:39 +0100 Subject: [PATCH 06/13] Expose XPackExtensions via SPI (elastic/x-pack-elasticsearch#3530) This change adds SPI loading for XPackExtensions that allows to extend XPack via an ordinary plugin. This can co-exist with the existin extension mechanism for the time being. Original commit: elastic/x-pack-elasticsearch@bf02b56deee7d1fccc1df4a96755bdbc4b69355c --- .../xpack/extensions/XPackExtension.java | 79 ++---------- .../xpack/security/SecurityExtension.java | 106 +++++++++++++++ .../xpack/security/Security.java | 27 ++-- .../xpack/security/SecurityTests.java | 13 +- .../build.gradle | 50 ++++++++ .../example/ExampleSecurityExtension.java | 67 ++++++++++ .../example/SpiExtensionPlugin.java | 31 +++++ .../CustomAuthenticationFailureHandler.java | 50 ++++++++ .../example/realm/CustomRealm.java | 70 ++++++++++ .../role/CustomInMemoryRolesProvider.java | 57 +++++++++ .../plugin-metadata/plugin-security.policy | 3 + ...ticsearch.xpack.security.SecurityExtension | 1 + .../example/realm/CustomRealmIT.java | 121 ++++++++++++++++++ .../example/realm/CustomRealmTests.java | 48 +++++++ .../example/role/CustomRolesProviderIT.java | 95 ++++++++++++++ 15 files changed, 730 insertions(+), 88 deletions(-) create mode 100644 plugin/core/src/main/java/org/elasticsearch/xpack/security/SecurityExtension.java create mode 100644 qa/security-example-spi-extension/build.gradle create mode 100644 qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/ExampleSecurityExtension.java create mode 100644 qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/SpiExtensionPlugin.java create mode 100644 qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/realm/CustomAuthenticationFailureHandler.java create mode 100644 qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/realm/CustomRealm.java create mode 100644 qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/role/CustomInMemoryRolesProvider.java create mode 100644 qa/security-example-spi-extension/src/main/plugin-metadata/plugin-security.policy create mode 100644 qa/security-example-spi-extension/src/main/resources/META-INF/services/org.elasticsearch.xpack.security.SecurityExtension create mode 100644 qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/realm/CustomRealmIT.java create mode 100644 qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/realm/CustomRealmTests.java create mode 100644 qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/role/CustomRolesProviderIT.java diff --git a/plugin/core/src/main/java/org/elasticsearch/xpack/extensions/XPackExtension.java b/plugin/core/src/main/java/org/elasticsearch/xpack/extensions/XPackExtension.java index c84de68b040..7ca3d65fd27 100644 --- a/plugin/core/src/main/java/org/elasticsearch/xpack/extensions/XPackExtension.java +++ b/plugin/core/src/main/java/org/elasticsearch/xpack/extensions/XPackExtension.java @@ -8,24 +8,16 @@ package org.elasticsearch.xpack.extensions; import java.util.Collection; import java.util.Collections; import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.function.BiConsumer; - -import org.elasticsearch.action.ActionListener; -import org.elasticsearch.common.settings.Setting; -import org.elasticsearch.common.settings.Settings; -import org.elasticsearch.watcher.ResourceWatcherService; -import org.elasticsearch.xpack.security.authc.AuthenticationFailureHandler; -import org.elasticsearch.xpack.security.authc.Realm; -import org.elasticsearch.xpack.security.authc.RealmConfig; -import org.elasticsearch.xpack.security.authz.RoleDescriptor; +import org.elasticsearch.plugins.Plugin; +import org.elasticsearch.xpack.security.SecurityExtension; /** * An extension point allowing to plug in custom functionality in x-pack authentication module. + * @deprecated use {@link SecurityExtension} via SPI instead */ -public abstract class XPackExtension { +@Deprecated +public abstract class XPackExtension implements SecurityExtension { /** * The name of the plugin. */ @@ -43,72 +35,21 @@ public abstract class XPackExtension { return Collections.emptyList(); } - /** - * Returns authentication realm implementations added by this extension. - * - * The key of the returned {@link Map} is the type name of the realm, and the value - * is a {@link org.elasticsearch.xpack.security.authc.Realm.Factory} which will construct - * that realm for use in authentication when that realm type is configured. - * - * @param resourceWatcherService Use to watch configuration files for changes - */ - public Map getRealms(ResourceWatcherService resourceWatcherService) { - return Collections.emptyMap(); - } - - /** - * Returns the set of {@link Setting settings} that may be configured for the each type of realm. - * - * Each setting key must be unqualified and is in the same format as will be provided via {@link RealmConfig#settings()}. - * If a given realm-type is not present in the returned map, then it will be treated as if it supported all possible settings. - * - * The life-cycle of an extension dictates that this method will be called before {@link #getRealms(ResourceWatcherService)} - */ - public Map>> getRealmSettings() { return Collections.emptyMap(); } - - /** - * Returns a handler for authentication failures, or null to use the default handler. - * - * Only one installed extension may have an authentication failure handler. If more than - * one extension returns a non-null handler, an error is raised. - */ - public AuthenticationFailureHandler getAuthenticationFailureHandler() { - return null; - } - /** * Returns a list of settings that should be filtered from API calls. In most cases, * these settings are sensitive such as passwords. * * The value should be the full name of the setting or a wildcard that matches the * desired setting. + * @deprecated use {@link Plugin#getSettingsFilter()} ()} via SPI extension instead */ + @Deprecated public List getSettingsFilter() { return Collections.emptyList(); } - /** - * Returns an ordered list of role providers that are used to resolve role names - * to {@link RoleDescriptor} objects. Each provider is invoked in order to - * resolve any role names not resolved by the reserved or native roles stores. - * - * Each role provider is represented as a {@link BiConsumer} which takes a set - * of roles to resolve as the first parameter to consume and an {@link ActionListener} - * as the second parameter to consume. The implementation of the role provider - * should be asynchronous if the computation is lengthy or any disk and/or network - * I/O is involved. The implementation is responsible for resolving whatever roles - * it can into a set of {@link RoleDescriptor} instances. If successful, the - * implementation must invoke {@link ActionListener#onResponse(Object)} to pass along - * the resolved set of role descriptors. If a failure was encountered, the - * implementation must invoke {@link ActionListener#onFailure(Exception)}. - * - * By default, an empty list is returned. - * - * @param settings The configured settings for the node - * @param resourceWatcherService Use to watch configuration files for changes - */ - public List, ActionListener>>> - getRolesProviders(Settings settings, ResourceWatcherService resourceWatcherService) { - return Collections.emptyList(); + @Override + public String toString() { + return name(); } } diff --git a/plugin/core/src/main/java/org/elasticsearch/xpack/security/SecurityExtension.java b/plugin/core/src/main/java/org/elasticsearch/xpack/security/SecurityExtension.java new file mode 100644 index 00000000000..6e0cde9f560 --- /dev/null +++ b/plugin/core/src/main/java/org/elasticsearch/xpack/security/SecurityExtension.java @@ -0,0 +1,106 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.xpack.security; + +import org.apache.lucene.util.SPIClassIterator; +import org.elasticsearch.action.ActionListener; +import org.elasticsearch.common.settings.Setting; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.watcher.ResourceWatcherService; +import org.elasticsearch.xpack.security.authc.AuthenticationFailureHandler; +import org.elasticsearch.xpack.security.authc.Realm; +import org.elasticsearch.xpack.security.authc.RealmConfig; +import org.elasticsearch.xpack.security.authz.RoleDescriptor; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.ServiceConfigurationError; +import java.util.Set; +import java.util.function.BiConsumer; + +/** + * An SPI extension point allowing to plug in custom functionality in x-pack authentication module. + */ +public interface SecurityExtension { + + /** + * Returns authentication realm implementations added by this extension. + * + * The key of the returned {@link Map} is the type name of the realm, and the value + * is a {@link Realm.Factory} which will construct + * that realm for use in authentication when that realm type is configured. + * + * @param resourceWatcherService Use to watch configuration files for changes + */ + default Map getRealms(ResourceWatcherService resourceWatcherService) { + return Collections.emptyMap(); + } + + /** + * Returns the set of {@link Setting settings} that may be configured for the each type of realm. + * + * Each setting key must be unqualified and is in the same format as will be provided via {@link RealmConfig#settings()}. + * If a given realm-type is not present in the returned map, then it will be treated as if it supported all possible settings. + * + * The life-cycle of an extension dictates that this method will be called before {@link #getRealms(ResourceWatcherService)} + */ + default Map>> getRealmSettings() { return Collections.emptyMap(); } + + /** + * Returns a handler for authentication failures, or null to use the default handler. + * + * Only one installed extension may have an authentication failure handler. If more than + * one extension returns a non-null handler, an error is raised. + */ + default AuthenticationFailureHandler getAuthenticationFailureHandler() { + return null; + } + + /** + * Returns an ordered list of role providers that are used to resolve role names + * to {@link RoleDescriptor} objects. Each provider is invoked in order to + * resolve any role names not resolved by the reserved or native roles stores. + * + * Each role provider is represented as a {@link BiConsumer} which takes a set + * of roles to resolve as the first parameter to consume and an {@link ActionListener} + * as the second parameter to consume. The implementation of the role provider + * should be asynchronous if the computation is lengthy or any disk and/or network + * I/O is involved. The implementation is responsible for resolving whatever roles + * it can into a set of {@link RoleDescriptor} instances. If successful, the + * implementation must invoke {@link ActionListener#onResponse(Object)} to pass along + * the resolved set of role descriptors. If a failure was encountered, the + * implementation must invoke {@link ActionListener#onFailure(Exception)}. + * + * By default, an empty list is returned. + * + * @param settings The configured settings for the node + * @param resourceWatcherService Use to watch configuration files for changes + */ + default List, ActionListener>>> + getRolesProviders(Settings settings, ResourceWatcherService resourceWatcherService) { + return Collections.emptyList(); + } + + /** + * Loads the XPackSecurityExtensions from the given class loader + */ + static List loadExtensions(ClassLoader loader) { + SPIClassIterator iterator = SPIClassIterator.get(SecurityExtension.class, loader); + List extensions = new ArrayList<>(); + while (iterator.hasNext()) { + final Class c = iterator.next(); + try { + extensions.add(c.getConstructor().newInstance()); + } catch (Exception e) { + throw new ServiceConfigurationError("failed to load security extension [" + c.getName() + "]", e); + } + } + return extensions; + } + +} diff --git a/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java b/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java index 06b4c8c3e76..32fe0a061d6 100644 --- a/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java +++ b/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java @@ -264,6 +264,8 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw private final SetOnce securityActionFilter = new SetOnce<>(); private final List bootstrapChecks; private final XPackExtensionsService extensionsService; + private final List securityExtensions = new ArrayList<>(); + public Security(Settings settings, final Path configPath) { this.settings = settings; @@ -350,15 +352,16 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw try { return createComponents(client, threadPool, clusterService, resourceWatcherService, - extensionsService.getExtensions()); + extensionsService.getExtensions().stream().collect(Collectors.toList())); } catch (final Exception e) { throw new IllegalStateException("security initialization failed", e); } } - public Collection createComponents(Client client, ThreadPool threadPool, ClusterService clusterService, + // pkg private for testing - tests want to pass in their set of extensions hence we are not using the extension service directly + Collection createComponents(Client client, ThreadPool threadPool, ClusterService clusterService, ResourceWatcherService resourceWatcherService, - List extensions) throws Exception { + List extensions) throws Exception { if (enabled == false) { return Collections.emptyList(); } @@ -411,7 +414,10 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw anonymousUser, securityLifecycleService, threadPool.getThreadContext()); Map realmFactories = new HashMap<>(InternalRealms.getFactories(threadPool, resourceWatcherService, getSslService(), nativeUsersStore, nativeRoleMappingStore, securityLifecycleService)); - for (XPackExtension extension : extensions) { + for (SecurityExtension extension : securityExtensions) { + extensions.add(extension); + } + for (SecurityExtension extension : extensions) { Map newRealms = extension.getRealms(resourceWatcherService); for (Map.Entry entry : newRealms.entrySet()) { if (realmFactories.put(entry.getKey(), entry.getValue()) != null) { @@ -427,14 +433,14 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw AuthenticationFailureHandler failureHandler = null; String extensionName = null; - for (XPackExtension extension : extensions) { + for (SecurityExtension extension : extensions) { AuthenticationFailureHandler extensionFailureHandler = extension.getAuthenticationFailureHandler(); if (extensionFailureHandler != null && failureHandler != null) { - throw new IllegalStateException("Extensions [" + extensionName + "] and [" + extension.name() + "] " + + throw new IllegalStateException("Extensions [" + extensionName + "] and [" + extension.toString() + "] " + "both set an authentication failure handler"); } failureHandler = extensionFailureHandler; - extensionName = extension.name(); + extensionName = extension.toString(); } if (failureHandler == null) { logger.debug("Using default authentication failure handler"); @@ -451,7 +457,7 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw final NativeRolesStore nativeRolesStore = new NativeRolesStore(settings, client, getLicenseState(), securityLifecycleService); final ReservedRolesStore reservedRolesStore = new ReservedRolesStore(); List, ActionListener>>> rolesProviders = new ArrayList<>(); - for (XPackExtension extension : extensions) { + for (SecurityExtension extension : extensions) { rolesProviders.addAll(extension.getRolesProviders(settings, resourceWatcherService)); } final CompositeRolesStore allRolesStore = new CompositeRolesStore(settings, fileRolesStore, nativeRolesStore, @@ -1043,4 +1049,9 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw } } } + + @Override + public void reloadSPI(ClassLoader loader) { + securityExtensions.addAll(SecurityExtension.loadExtensions(loader)); + } } diff --git a/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecurityTests.java b/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecurityTests.java index 08969555cbb..73722ad1ec6 100644 --- a/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecurityTests.java +++ b/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecurityTests.java @@ -31,7 +31,6 @@ import org.elasticsearch.test.ESTestCase; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.watcher.ResourceWatcherService; import org.elasticsearch.xpack.XPackSettings; -import org.elasticsearch.xpack.extensions.XPackExtension; import org.elasticsearch.xpack.security.audit.AuditTrailService; import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail; import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail; @@ -75,26 +74,18 @@ public class SecurityTests extends ESTestCase { private ThreadContext threadContext = null; private TestUtils.UpdatableLicenseState licenseState; - public static class DummyExtension extends XPackExtension { + public static class DummyExtension implements SecurityExtension { private String realmType; DummyExtension(String realmType) { this.realmType = realmType; } @Override - public String name() { - return "dummy"; - } - @Override - public String description() { - return "dummy"; - } - @Override public Map getRealms(ResourceWatcherService resourceWatcherService) { return Collections.singletonMap(realmType, config -> null); } } - private Collection createComponents(Settings testSettings, XPackExtension... extensions) throws Exception { + private Collection createComponents(Settings testSettings, SecurityExtension... extensions) throws Exception { if (security != null) { throw new IllegalStateException("Security object already exists (" + security + ")"); } diff --git a/qa/security-example-spi-extension/build.gradle b/qa/security-example-spi-extension/build.gradle new file mode 100644 index 00000000000..f6e5e44398a --- /dev/null +++ b/qa/security-example-spi-extension/build.gradle @@ -0,0 +1,50 @@ +apply plugin: 'elasticsearch.esplugin' + +esplugin { + name 'spi-extension' + description 'An example spi extension pluing for xpack security' + classname 'org.elasticsearch.example.SpiExtensionPlugin' + extendedPlugins = ['x-pack-security'] +} + +dependencies { + provided project(path: ':x-pack-elasticsearch:plugin:core', configuration: 'runtime') + testCompile project(path: ':x-pack-elasticsearch:transport-client', configuration: 'runtime') +} + + +integTestRunner { + systemProperty 'tests.security.manager', 'false' +} + +integTestCluster { + dependsOn buildZip + plugin ':x-pack-elasticsearch:plugin' + setting 'xpack.security.authc.realms.custom.order', '0' + setting 'xpack.security.authc.realms.custom.type', 'custom' + setting 'xpack.security.authc.realms.custom.filtered_setting', 'should be filtered' + setting 'xpack.security.authc.realms.esusers.order', '1' + setting 'xpack.security.authc.realms.esusers.type', 'file' + setting 'xpack.security.authc.realms.native.type', 'native' + setting 'xpack.security.authc.realms.native.order', '2' + setting 'xpack.ml.enabled', 'false' + + // This is important, so that all the modules are available too. + // There are index templates that use token filters that are in analysis-module and + // processors are being used that are in ingest-common module. + distribution = 'zip' + + setupCommand 'setupDummyUser', + 'bin/x-pack/users', 'useradd', 'test_user', '-p', 'x-pack-test-password', '-r', 'superuser' + waitCondition = { node, ant -> + File tmpFile = new File(node.cwd, 'wait.success') + ant.get(src: "http://${node.httpUri()}/_cluster/health?wait_for_nodes=>=${numNodes}&wait_for_status=yellow", + dest: tmpFile.toString(), + username: 'test_user', + password: 'x-pack-test-password', + ignoreerrors: true, + retries: 10) + return tmpFile.exists() + } +} +check.dependsOn integTest diff --git a/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/ExampleSecurityExtension.java b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/ExampleSecurityExtension.java new file mode 100644 index 00000000000..9bae1a03816 --- /dev/null +++ b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/ExampleSecurityExtension.java @@ -0,0 +1,67 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.example; + +import org.elasticsearch.action.ActionListener; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.example.realm.CustomAuthenticationFailureHandler; +import org.elasticsearch.example.realm.CustomRealm; +import org.elasticsearch.example.role.CustomInMemoryRolesProvider; +import org.elasticsearch.watcher.ResourceWatcherService; +import org.elasticsearch.xpack.security.SecurityExtension; +import org.elasticsearch.xpack.security.authc.AuthenticationFailureHandler; +import org.elasticsearch.xpack.security.authc.Realm; +import org.elasticsearch.xpack.security.authz.RoleDescriptor; + +import java.security.AccessController; +import java.security.PrivilegedAction; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.BiConsumer; + +import static org.elasticsearch.example.role.CustomInMemoryRolesProvider.ROLE_A; +import static org.elasticsearch.example.role.CustomInMemoryRolesProvider.ROLE_B; + +/** + * An example x-pack extension for testing custom realms and custom role providers. + */ +public class ExampleSecurityExtension implements SecurityExtension { + + static { + // check that the extension's policy works. + AccessController.doPrivileged((PrivilegedAction) () -> { + System.getSecurityManager().checkPrintJobAccess(); + return null; + }); + } + + @Override + public Map getRealms(ResourceWatcherService resourceWatcherService) { + return Collections.singletonMap(CustomRealm.TYPE, CustomRealm::new); + } + + @Override + public AuthenticationFailureHandler getAuthenticationFailureHandler() { + return new CustomAuthenticationFailureHandler(); + } + + + @Override + public List, ActionListener>>> + getRolesProviders(Settings settings, ResourceWatcherService resourceWatcherService) { + CustomInMemoryRolesProvider rp1 = new CustomInMemoryRolesProvider(settings, Collections.singletonMap(ROLE_A, "read")); + Map roles = new HashMap<>(); + roles.put(ROLE_A, "all"); + roles.put(ROLE_B, "all"); + CustomInMemoryRolesProvider rp2 = new CustomInMemoryRolesProvider(settings, roles); + return Arrays.asList(rp1, rp2); + } +} diff --git a/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/SpiExtensionPlugin.java b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/SpiExtensionPlugin.java new file mode 100644 index 00000000000..07f769849d5 --- /dev/null +++ b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/SpiExtensionPlugin.java @@ -0,0 +1,31 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.example; + +import org.elasticsearch.example.realm.CustomRealm; +import org.elasticsearch.plugins.ActionPlugin; +import org.elasticsearch.plugins.Plugin; + +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.List; + +/** + * An example x-pack extension for testing custom realms and custom role providers. + */ +public class SpiExtensionPlugin extends Plugin implements ActionPlugin { + + @Override + public Collection getRestHeaders() { + return Arrays.asList(CustomRealm.USER_HEADER, CustomRealm.PW_HEADER); + } + + @Override + public List getSettingsFilter() { + return Collections.singletonList("xpack.security.authc.realms.*.filtered_setting"); + } +} diff --git a/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/realm/CustomAuthenticationFailureHandler.java b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/realm/CustomAuthenticationFailureHandler.java new file mode 100644 index 00000000000..ecc2567e0c1 --- /dev/null +++ b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/realm/CustomAuthenticationFailureHandler.java @@ -0,0 +1,50 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.example.realm; + +import org.elasticsearch.ElasticsearchSecurityException; +import org.elasticsearch.common.util.concurrent.ThreadContext; +import org.elasticsearch.rest.RestRequest; +import org.elasticsearch.xpack.security.authc.AuthenticationToken; +import org.elasticsearch.xpack.security.authc.DefaultAuthenticationFailureHandler; +import org.elasticsearch.transport.TransportMessage; + +public class CustomAuthenticationFailureHandler extends DefaultAuthenticationFailureHandler { + + @Override + public ElasticsearchSecurityException failedAuthentication(RestRequest request, AuthenticationToken token, + ThreadContext context) { + ElasticsearchSecurityException e = super.failedAuthentication(request, token, context); + // set a custom header + e.addHeader("WWW-Authenticate", "custom-challenge"); + return e; + } + + @Override + public ElasticsearchSecurityException failedAuthentication(TransportMessage message, AuthenticationToken token, String action, + ThreadContext context) { + ElasticsearchSecurityException e = super.failedAuthentication(message, token, action, context); + // set a custom header + e.addHeader("WWW-Authenticate", "custom-challenge"); + return e; + } + + @Override + public ElasticsearchSecurityException missingToken(RestRequest request, ThreadContext context) { + ElasticsearchSecurityException e = super.missingToken(request, context); + // set a custom header + e.addHeader("WWW-Authenticate", "custom-challenge"); + return e; + } + + @Override + public ElasticsearchSecurityException missingToken(TransportMessage message, String action, ThreadContext context) { + ElasticsearchSecurityException e = super.missingToken(message, action, context); + // set a custom header + e.addHeader("WWW-Authenticate", "custom-challenge"); + return e; + } +} diff --git a/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/realm/CustomRealm.java b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/realm/CustomRealm.java new file mode 100644 index 00000000000..d110a6ca071 --- /dev/null +++ b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/realm/CustomRealm.java @@ -0,0 +1,70 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.example.realm; + +import org.elasticsearch.action.ActionListener; +import org.elasticsearch.common.settings.SecureString; +import org.elasticsearch.common.util.concurrent.ThreadContext; +import org.elasticsearch.xpack.security.authc.AuthenticationResult; +import org.elasticsearch.xpack.security.authc.AuthenticationToken; +import org.elasticsearch.xpack.security.authc.Realm; +import org.elasticsearch.xpack.security.authc.RealmConfig; +import org.elasticsearch.xpack.security.authc.support.CharArrays; +import org.elasticsearch.xpack.security.authc.support.UsernamePasswordToken; +import org.elasticsearch.xpack.security.user.User; + +public class CustomRealm extends Realm { + + public static final String TYPE = "custom"; + + public static final String USER_HEADER = "User"; + public static final String PW_HEADER = "Password"; + + public static final String KNOWN_USER = "custom_user"; + public static final SecureString KNOWN_PW = new SecureString("x-pack-test-password".toCharArray()); + static final String[] ROLES = new String[] { "superuser" }; + + public CustomRealm(RealmConfig config) { + super(TYPE, config); + } + + @Override + public boolean supports(AuthenticationToken token) { + return token instanceof UsernamePasswordToken; + } + + @Override + public UsernamePasswordToken token(ThreadContext threadContext) { + String user = threadContext.getHeader(USER_HEADER); + if (user != null) { + String password = threadContext.getHeader(PW_HEADER); + if (password != null) { + return new UsernamePasswordToken(user, new SecureString(password.toCharArray())); + } + } + return null; + } + + @Override + public void authenticate(AuthenticationToken authToken, ActionListener listener) { + UsernamePasswordToken token = (UsernamePasswordToken)authToken; + final String actualUser = token.principal(); + if (KNOWN_USER.equals(actualUser)) { + if (CharArrays.constantTimeEquals(token.credentials().getChars(), KNOWN_PW.getChars())) { + listener.onResponse(AuthenticationResult.success(new User(actualUser, ROLES))); + } else { + listener.onResponse(AuthenticationResult.unsuccessful("Invalid password for user " + actualUser, null)); + } + } else { + listener.onResponse(AuthenticationResult.notHandled()); + } + } + + @Override + public void lookupUser(String username, ActionListener listener) { + listener.onResponse(null); + } +} diff --git a/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/role/CustomInMemoryRolesProvider.java b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/role/CustomInMemoryRolesProvider.java new file mode 100644 index 00000000000..e1f4680c6c9 --- /dev/null +++ b/qa/security-example-spi-extension/src/main/java/org/elasticsearch/example/role/CustomInMemoryRolesProvider.java @@ -0,0 +1,57 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.example.role; + +import org.elasticsearch.action.ActionListener; +import org.elasticsearch.common.component.AbstractComponent; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.xpack.security.authz.RoleDescriptor; + +import java.util.HashSet; +import java.util.Map; +import java.util.Set; +import java.util.function.BiConsumer; + +/** + * A custom roles provider implementation for testing that serves + * static roles from memory. + */ +public class CustomInMemoryRolesProvider + extends AbstractComponent + implements BiConsumer, ActionListener>> { + + public static final String INDEX = "foo"; + public static final String ROLE_A = "roleA"; + public static final String ROLE_B = "roleB"; + + private final Map rolePermissionSettings; + + public CustomInMemoryRolesProvider(Settings settings, Map rolePermissionSettings) { + super(settings); + this.rolePermissionSettings = rolePermissionSettings; + } + + @Override + public void accept(Set roles, ActionListener> listener) { + Set roleDescriptors = new HashSet<>(); + for (String role : roles) { + if (rolePermissionSettings.containsKey(role)) { + roleDescriptors.add( + new RoleDescriptor(role, new String[] { "all" }, + new RoleDescriptor.IndicesPrivileges[] { + RoleDescriptor.IndicesPrivileges.builder() + .privileges(rolePermissionSettings.get(role)) + .indices(INDEX) + .grantedFields("*") + .build() + }, null) + ); + } + } + + listener.onResponse(roleDescriptors); + } +} diff --git a/qa/security-example-spi-extension/src/main/plugin-metadata/plugin-security.policy b/qa/security-example-spi-extension/src/main/plugin-metadata/plugin-security.policy new file mode 100644 index 00000000000..a3b647b316e --- /dev/null +++ b/qa/security-example-spi-extension/src/main/plugin-metadata/plugin-security.policy @@ -0,0 +1,3 @@ +grant { + permission java.lang.RuntimePermission "queuePrintJob"; +}; diff --git a/qa/security-example-spi-extension/src/main/resources/META-INF/services/org.elasticsearch.xpack.security.SecurityExtension b/qa/security-example-spi-extension/src/main/resources/META-INF/services/org.elasticsearch.xpack.security.SecurityExtension new file mode 100644 index 00000000000..cba4e93a292 --- /dev/null +++ b/qa/security-example-spi-extension/src/main/resources/META-INF/services/org.elasticsearch.xpack.security.SecurityExtension @@ -0,0 +1 @@ +org.elasticsearch.example.ExampleSecurityExtension \ No newline at end of file diff --git a/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/realm/CustomRealmIT.java b/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/realm/CustomRealmIT.java new file mode 100644 index 00000000000..da6d62b1bf0 --- /dev/null +++ b/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/realm/CustomRealmIT.java @@ -0,0 +1,121 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.example.realm; + +import org.apache.http.message.BasicHeader; +import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse; +import org.elasticsearch.action.admin.cluster.node.info.NodeInfo; +import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse; +import org.elasticsearch.client.Response; +import org.elasticsearch.client.ResponseException; +import org.elasticsearch.client.transport.NoNodeAvailableException; +import org.elasticsearch.client.transport.TransportClient; +import org.elasticsearch.common.network.NetworkModule; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.transport.TransportAddress; +import org.elasticsearch.common.util.concurrent.ThreadContext; +import org.elasticsearch.env.Environment; +import org.elasticsearch.plugins.Plugin; +import org.elasticsearch.test.ESIntegTestCase; +import org.elasticsearch.xpack.XPackClientPlugin; +import org.elasticsearch.xpack.client.PreBuiltXPackTransportClient; + +import java.util.Collection; +import java.util.Collections; +import java.util.List; + +import static org.hamcrest.Matchers.is; + +/** + * Integration test to test authentication with the custom realm + */ +public class CustomRealmIT extends ESIntegTestCase { + + @Override + protected Settings externalClusterClientSettings() { + return Settings.builder() + .put(ThreadContext.PREFIX + "." + CustomRealm.USER_HEADER, CustomRealm.KNOWN_USER) + .put(ThreadContext.PREFIX + "." + CustomRealm.PW_HEADER, CustomRealm.KNOWN_PW.toString()) + .put(NetworkModule.TRANSPORT_TYPE_KEY, "security4") + .build(); + } + + @Override + protected Collection> transportClientPlugins() { + return Collections.>singleton(XPackClientPlugin.class); + } + + public void testHttpConnectionWithNoAuthentication() throws Exception { + try { + getRestClient().performRequest("GET", "/"); + fail("request should have failed"); + } catch(ResponseException e) { + Response response = e.getResponse(); + assertThat(response.getStatusLine().getStatusCode(), is(401)); + String value = response.getHeader("WWW-Authenticate"); + assertThat(value, is("custom-challenge")); + } + } + + public void testHttpAuthentication() throws Exception { + Response response = getRestClient().performRequest("GET", "/", + new BasicHeader(CustomRealm.USER_HEADER, CustomRealm.KNOWN_USER), + new BasicHeader(CustomRealm.PW_HEADER, CustomRealm.KNOWN_PW.toString())); + assertThat(response.getStatusLine().getStatusCode(), is(200)); + } + + public void testTransportClient() throws Exception { + NodesInfoResponse nodeInfos = client().admin().cluster().prepareNodesInfo().get(); + List nodes = nodeInfos.getNodes(); + assertTrue(nodes.isEmpty() == false); + TransportAddress publishAddress = randomFrom(nodes).getTransport().address().publishAddress(); + String clusterName = nodeInfos.getClusterName().value(); + + Settings settings = Settings.builder() + .put("cluster.name", clusterName) + .put(Environment.PATH_HOME_SETTING.getKey(), createTempDir().toAbsolutePath().toString()) + .put(ThreadContext.PREFIX + "." + CustomRealm.USER_HEADER, CustomRealm.KNOWN_USER) + .put(ThreadContext.PREFIX + "." + CustomRealm.PW_HEADER, CustomRealm.KNOWN_PW.toString()) + .build(); + try (TransportClient client = new PreBuiltXPackTransportClient(settings)) { + client.addTransportAddress(publishAddress); + ClusterHealthResponse response = client.admin().cluster().prepareHealth().execute().actionGet(); + assertThat(response.isTimedOut(), is(false)); + } + } + + public void testTransportClientWrongAuthentication() throws Exception { + NodesInfoResponse nodeInfos = client().admin().cluster().prepareNodesInfo().get(); + List nodes = nodeInfos.getNodes(); + assertTrue(nodes.isEmpty() == false); + TransportAddress publishAddress = randomFrom(nodes).getTransport().address().publishAddress(); + String clusterName = nodeInfos.getClusterName().value(); + + Settings settings = Settings.builder() + .put("cluster.name", clusterName) + .put(Environment.PATH_HOME_SETTING.getKey(), createTempDir().toAbsolutePath().toString()) + .put(ThreadContext.PREFIX + "." + CustomRealm.USER_HEADER, CustomRealm.KNOWN_USER + randomAlphaOfLength(1)) + .put(ThreadContext.PREFIX + "." + CustomRealm.PW_HEADER, CustomRealm.KNOWN_PW.toString()) + .build(); + try (TransportClient client = new PreBuiltXPackTransportClient(settings)) { + client.addTransportAddress(publishAddress); + client.admin().cluster().prepareHealth().execute().actionGet(); + fail("authentication failure should have resulted in a NoNodesAvailableException"); + } catch (NoNodeAvailableException e) { + // expected + } + } + + public void testSettingsFiltering() throws Exception { + NodesInfoResponse nodeInfos = client().admin().cluster().prepareNodesInfo().clear().setSettings(true).get(); + for(NodeInfo info : nodeInfos.getNodes()) { + Settings settings = info.getSettings(); + assertNotNull(settings); + assertNull(settings.get("xpack.security.authc.realms.custom.filtered_setting")); + assertEquals(CustomRealm.TYPE, settings.get("xpack.security.authc.realms.custom.type")); + } + } +} diff --git a/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/realm/CustomRealmTests.java b/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/realm/CustomRealmTests.java new file mode 100644 index 00000000000..c0532d99ab6 --- /dev/null +++ b/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/realm/CustomRealmTests.java @@ -0,0 +1,48 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.example.realm; + +import org.elasticsearch.action.support.PlainActionFuture; +import org.elasticsearch.common.settings.SecureString; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.util.concurrent.ThreadContext; +import org.elasticsearch.env.TestEnvironment; +import org.elasticsearch.test.ESTestCase; +import org.elasticsearch.xpack.security.authc.AuthenticationResult; +import org.elasticsearch.xpack.security.authc.RealmConfig; +import org.elasticsearch.xpack.security.authc.support.UsernamePasswordToken; +import org.elasticsearch.xpack.security.user.User; + +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.notNullValue; + +public class CustomRealmTests extends ESTestCase { + public void testAuthenticate() { + Settings globalSettings = Settings.builder().put("path.home", createTempDir()).build(); + CustomRealm realm = new CustomRealm(new RealmConfig("test", Settings.EMPTY, globalSettings, + TestEnvironment.newEnvironment(globalSettings), new ThreadContext(globalSettings))); + SecureString password = CustomRealm.KNOWN_PW.clone(); + UsernamePasswordToken token = new UsernamePasswordToken(CustomRealm.KNOWN_USER, password); + PlainActionFuture plainActionFuture = new PlainActionFuture<>(); + realm.authenticate(token, plainActionFuture); + User user = plainActionFuture.actionGet().getUser(); + assertThat(user, notNullValue()); + assertThat(user.roles(), equalTo(CustomRealm.ROLES)); + assertThat(user.principal(), equalTo(CustomRealm.KNOWN_USER)); + } + + public void testAuthenticateBadUser() { + Settings globalSettings = Settings.builder().put("path.home", createTempDir()).build(); + CustomRealm realm = new CustomRealm(new RealmConfig("test", Settings.EMPTY, globalSettings, + TestEnvironment.newEnvironment(globalSettings), new ThreadContext(globalSettings))); + SecureString password = CustomRealm.KNOWN_PW.clone(); + UsernamePasswordToken token = new UsernamePasswordToken(CustomRealm.KNOWN_USER + "1", password); + PlainActionFuture plainActionFuture = new PlainActionFuture<>(); + realm.authenticate(token, plainActionFuture); + final AuthenticationResult result = plainActionFuture.actionGet(); + assertThat(result.getStatus(), equalTo(AuthenticationResult.Status.CONTINUE)); + } +} diff --git a/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/role/CustomRolesProviderIT.java b/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/role/CustomRolesProviderIT.java new file mode 100644 index 00000000000..eeccf0c7b4e --- /dev/null +++ b/qa/security-example-spi-extension/src/test/java/org/elasticsearch/example/role/CustomRolesProviderIT.java @@ -0,0 +1,95 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.example.role; + +import org.apache.http.message.BasicHeader; +import org.elasticsearch.client.Response; +import org.elasticsearch.client.ResponseException; +import org.elasticsearch.common.network.NetworkModule; +import org.elasticsearch.common.settings.SecureString; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.util.concurrent.ThreadContext; +import org.elasticsearch.example.realm.CustomRealm; +import org.elasticsearch.plugins.Plugin; +import org.elasticsearch.test.ESIntegTestCase; +import org.elasticsearch.xpack.XPackClientPlugin; +import org.elasticsearch.xpack.security.authc.support.UsernamePasswordToken; +import org.elasticsearch.xpack.security.client.SecurityClient; + +import java.util.Collection; +import java.util.Collections; + +import static org.elasticsearch.example.role.CustomInMemoryRolesProvider.INDEX; +import static org.elasticsearch.example.role.CustomInMemoryRolesProvider.ROLE_A; +import static org.elasticsearch.example.role.CustomInMemoryRolesProvider.ROLE_B; +import static org.elasticsearch.xpack.security.authc.support.UsernamePasswordToken.basicAuthHeaderValue; +import static org.hamcrest.Matchers.is; + +/** + * Integration test for custom roles providers. + */ +public class CustomRolesProviderIT extends ESIntegTestCase { + + private static final String TEST_USER = "test_user"; + private static final String TEST_PWD = "change_me"; + + @Override + protected Settings externalClusterClientSettings() { + return Settings.builder() + .put(ThreadContext.PREFIX + "." + CustomRealm.USER_HEADER, CustomRealm.KNOWN_USER) + .put(ThreadContext.PREFIX + "." + CustomRealm.PW_HEADER, CustomRealm.KNOWN_PW.toString()) + .put(NetworkModule.TRANSPORT_TYPE_KEY, "security4") + .build(); + } + + @Override + protected Collection> transportClientPlugins() { + return Collections.singleton(XPackClientPlugin.class); + } + + public void setupTestUser(String role) { + SecurityClient securityClient = new SecurityClient(client()); + securityClient.preparePutUser(TEST_USER, TEST_PWD.toCharArray(), role).get(); + } + + public void testAuthorizedCustomRoleSucceeds() throws Exception { + setupTestUser(ROLE_B); + // roleB has all permissions on index "foo", so creating "foo" should succeed + Response response = getRestClient().performRequest("PUT", "/" + INDEX, authHeader()); + assertThat(response.getStatusLine().getStatusCode(), is(200)); + } + + public void testFirstResolvedRoleTakesPrecedence() throws Exception { + // the first custom roles provider has set ROLE_A to only have read permission on the index, + // the second custom roles provider has set ROLE_A to have all permissions, but since + // the first custom role provider appears first in order, it should take precedence and deny + // permission to create the index + setupTestUser(ROLE_A); + // roleB has all permissions on index "foo", so creating "foo" should succeed + try { + getRestClient().performRequest("PUT", "/" + INDEX, authHeader()); + fail(ROLE_A + " should not be authorized to create index " + INDEX); + } catch (ResponseException e) { + assertThat(e.getResponse().getStatusLine().getStatusCode(), is(403)); + } + } + + public void testUnresolvedRoleDoesntSucceed() throws Exception { + setupTestUser("unknown"); + // roleB has all permissions on index "foo", so creating "foo" should succeed + try { + getRestClient().performRequest("PUT", "/" + INDEX, authHeader()); + fail(ROLE_A + " should not be authorized to create index " + INDEX); + } catch (ResponseException e) { + assertThat(e.getResponse().getStatusLine().getStatusCode(), is(403)); + } + } + + private BasicHeader authHeader() { + return new BasicHeader(UsernamePasswordToken.BASIC_AUTH_HEADER, + basicAuthHeaderValue(TEST_USER, new SecureString(TEST_PWD.toCharArray()))); + } +} From e3da8fa4aefa0f2330b06991b79a2f3aaf19ff95 Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Tue, 23 Jan 2018 14:13:22 +0100 Subject: [PATCH 07/13] Cleanup: Remove HaltedClock (elastic/x-pack-elasticsearch#3664) The HaltedClock was a leftover from moving over from our own Clock implementation to a java.time one. java.time already has a fixed clock, this one is not needed. Original commit: elastic/x-pack-elasticsearch@f91c401a60c28d2c22bf1a3c5c206988cf2d9e1b --- .../xpack/common/time/HaltedClock.java | 47 ------------------- .../SamlLogoutRequestMessageBuilderTests.java | 11 ++--- .../xpack/watcher/watch/WatchParser.java | 8 ++-- 3 files changed, 10 insertions(+), 56 deletions(-) delete mode 100644 plugin/core/src/main/java/org/elasticsearch/xpack/common/time/HaltedClock.java diff --git a/plugin/core/src/main/java/org/elasticsearch/xpack/common/time/HaltedClock.java b/plugin/core/src/main/java/org/elasticsearch/xpack/common/time/HaltedClock.java deleted file mode 100644 index 809a737abb6..00000000000 --- a/plugin/core/src/main/java/org/elasticsearch/xpack/common/time/HaltedClock.java +++ /dev/null @@ -1,47 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License; - * you may not use this file except in compliance with the Elastic License. - */ -package org.elasticsearch.xpack.common.time; - -import org.joda.time.DateTime; -import org.joda.time.DateTimeZone; - -import java.time.Clock; -import java.time.Instant; -import java.time.ZoneId; -import java.time.ZoneOffset; - -public class HaltedClock extends Clock { - - private final DateTime now; - - public HaltedClock(DateTime now) { - this.now = now.toDateTime(DateTimeZone.UTC); - } - - @Override - public ZoneId getZone() { - return ZoneOffset.UTC; - } - - @Override - public Clock withZone(ZoneId zoneId) { - if (zoneId.equals(ZoneOffset.UTC)) { - return this; - } - - throw new IllegalArgumentException("Halted clock time zone cannot be changed"); - } - - @Override - public long millis() { - return now.getMillis(); - } - - @Override - public Instant instant() { - return Instant.ofEpochMilli(now.getMillis()); - } -} diff --git a/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlLogoutRequestMessageBuilderTests.java b/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlLogoutRequestMessageBuilderTests.java index 88d574de489..252cf2f0d1c 100644 --- a/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlLogoutRequestMessageBuilderTests.java +++ b/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlLogoutRequestMessageBuilderTests.java @@ -6,11 +6,10 @@ package org.elasticsearch.xpack.security.authc.saml; import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; -import org.elasticsearch.xpack.common.time.HaltedClock; import org.hamcrest.Matchers; -import org.joda.time.DateTime; -import org.joda.time.DateTimeZone; import org.junit.Before; import org.opensaml.saml.common.xml.SAMLConstants; import org.opensaml.saml.saml2.core.LogoutRequest; @@ -67,8 +66,8 @@ public class SamlLogoutRequestMessageBuilderTests extends SamlTestCase { "http://idp.example.com/saml/logout/artifact"); idpRole.getSingleLogoutServices().add(sloArtifact); - final DateTime now = DateTime.now(DateTimeZone.UTC); - final SamlLogoutRequestMessageBuilder builder = new SamlLogoutRequestMessageBuilder(new HaltedClock(now), sp, idp, nameId, session); + Clock fixedClock = Clock.fixed(Instant.now(), ZoneOffset.UTC); + final SamlLogoutRequestMessageBuilder builder = new SamlLogoutRequestMessageBuilder(fixedClock, sp, idp, nameId, session); final LogoutRequest logoutRequest = builder.build(); assertThat(logoutRequest, notNullValue()); assertThat(logoutRequest.getReason(), nullValue()); @@ -82,7 +81,7 @@ public class SamlLogoutRequestMessageBuilderTests extends SamlTestCase { assertThat(logoutRequest.getConsent(), nullValue()); assertThat(logoutRequest.getNotOnOrAfter(), nullValue()); assertThat(logoutRequest.getIssueInstant(), notNullValue()); - assertThat(logoutRequest.getIssueInstant(), equalTo(now)); + assertThat(logoutRequest.getIssueInstant().getMillis(), equalTo(fixedClock.millis())); assertThat(logoutRequest.getSessionIndexes(), iterableWithSize(1)); assertThat(logoutRequest.getSessionIndexes().get(0).getSessionIndex(), equalTo(session)); assertThat(logoutRequest.getDestination(), equalTo("http://idp.example.com/saml/logout/redirect")); diff --git a/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java b/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java index 7a8dda3fc86..bb68d1217ae 100644 --- a/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java +++ b/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java @@ -29,11 +29,12 @@ import org.elasticsearch.xpack.watcher.support.xcontent.WatcherXContentParser; import org.elasticsearch.xpack.watcher.transform.ExecutableTransform; import org.elasticsearch.xpack.watcher.trigger.Trigger; import org.elasticsearch.xpack.watcher.trigger.TriggerService; -import org.elasticsearch.xpack.common.time.HaltedClock; import org.joda.time.DateTime; import java.io.IOException; import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; import java.util.Collections; import java.util.HashMap; import java.util.List; @@ -102,8 +103,9 @@ public class WatchParser extends AbstractComponent { XContentParser parser = null; try { // EMPTY is safe here because we never use namedObject - parser = new WatcherXContentParser(xContentType.xContent().createParser(NamedXContentRegistry.EMPTY, source), - new HaltedClock(now), withSecrets ? cryptoService : null); + Clock fixedClock = Clock.fixed(Instant.now(), ZoneOffset.UTC); + parser = new WatcherXContentParser(xContentType.xContent().createParser(NamedXContentRegistry.EMPTY, source), fixedClock, + withSecrets ? cryptoService : null); parser.nextToken(); return parse(id, includeStatus, parser); } catch (IOException ioe) { From 215f9af1cc9c349b6eaf9d92725dc419781daece Mon Sep 17 00:00:00 2001 From: Dimitris Athanasiou Date: Tue, 23 Jan 2018 14:16:34 +0000 Subject: [PATCH 08/13] [ML] Add trace logging for when search response is obtained (elastic/x-pack-elasticsearch#3669) This is useful for understanding performance characteristics as it helps us understand whether the bottleneck is the search part or the analytics part. Relates elastic/x-pack-elasticsearch#3590 Original commit: elastic/x-pack-elasticsearch@dc8c095958fb057c75afccb0d5118ad676cfffcb --- .../extractor/aggregation/AggregationDataExtractor.java | 2 +- .../ml/datafeed/extractor/chunked/ChunkedDataExtractor.java | 1 + .../xpack/ml/datafeed/extractor/scroll/ScrollDataExtractor.java | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/AggregationDataExtractor.java b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/AggregationDataExtractor.java index 61cef8e9643..aa16f839d31 100644 --- a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/AggregationDataExtractor.java +++ b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/AggregationDataExtractor.java @@ -99,8 +99,8 @@ class AggregationDataExtractor implements DataExtractor { private Aggregations search() throws IOException { LOGGER.debug("[{}] Executing aggregated search", context.jobId); - SearchResponse searchResponse = executeSearchRequest(buildSearchRequest()); + LOGGER.debug("[{}] Search response was obtained", context.jobId); ExtractorUtils.checkSearchWasSuccessful(context.jobId, searchResponse); return validateAggs(searchResponse.getAggregations()); } diff --git a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/chunked/ChunkedDataExtractor.java b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/chunked/ChunkedDataExtractor.java index 158fdfa8c58..3e4eb0866d4 100644 --- a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/chunked/ChunkedDataExtractor.java +++ b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/chunked/ChunkedDataExtractor.java @@ -117,6 +117,7 @@ public class ChunkedDataExtractor implements DataExtractor { .addAggregation(AggregationBuilders.max(LATEST_TIME).field(context.timeField)); SearchResponse response = executeSearchRequest(searchRequestBuilder); + LOGGER.debug("[{}] Data summary response was obtained", context.jobId); ExtractorUtils.checkSearchWasSuccessful(context.jobId, response); diff --git a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/scroll/ScrollDataExtractor.java b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/scroll/ScrollDataExtractor.java index c0a6cb3df54..e64c4b0116e 100644 --- a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/scroll/ScrollDataExtractor.java +++ b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/scroll/ScrollDataExtractor.java @@ -95,6 +95,7 @@ class ScrollDataExtractor implements DataExtractor { protected InputStream initScroll(long startTimestamp) throws IOException { LOGGER.debug("[{}] Initializing scroll", context.jobId); SearchResponse searchResponse = executeSearchRequest(buildSearchRequest(startTimestamp)); + LOGGER.debug("[{}] Search response was obtained", context.jobId); return processSearchResponse(searchResponse); } @@ -195,6 +196,7 @@ class ScrollDataExtractor implements DataExtractor { throw searchExecutionException; } } + LOGGER.debug("[{}] Search response was obtained", context.jobId); return processSearchResponse(searchResponse); } From 63c0e288af5df14c9a006fde760a2b815568e5d4 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 23 Jan 2018 17:02:37 +0100 Subject: [PATCH 09/13] Ensure we protect Collections obtained from scripts from self-referencing (elastic/x-pack-elasticsearch#3681) Self referencing maps can cause SOE if they are iterated ie. in their toString methods. This chance adds some protected to the usage of those collections. see elastic/elasticsearch#28335 Original commit: elastic/x-pack-elasticsearch@c4f1089c74ab05f32f316395b4f513da4dbf24f1 --- .../execute/TransportExecuteWatchAction.java | 13 +++- .../test/watcher_painless/40_exception.yml | 74 +++++++++++++++++++ 2 files changed, 83 insertions(+), 4 deletions(-) diff --git a/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/transport/actions/execute/TransportExecuteWatchAction.java b/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/transport/actions/execute/TransportExecuteWatchAction.java index bc8b9f4e1b2..471424d792c 100644 --- a/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/transport/actions/execute/TransportExecuteWatchAction.java +++ b/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/transport/actions/execute/TransportExecuteWatchAction.java @@ -16,6 +16,7 @@ import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver; import org.elasticsearch.cluster.routing.Preference; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.util.concurrent.AbstractRunnable; import org.elasticsearch.common.xcontent.XContentBuilder; import org.elasticsearch.common.xcontent.XContentFactory; import org.elasticsearch.common.xcontent.XContentType; @@ -109,8 +110,14 @@ public class TransportExecuteWatchAction extends WatcherTransportAction listener, Watch watch, boolean knownWatch) { - threadPool.executor(XPackField.WATCHER).submit(() -> { - try { + threadPool.executor(XPackField.WATCHER).submit(new AbstractRunnable() { + @Override + public void onFailure(Exception e) { + listener.onFailure(e); + } + + @Override + protected void doRun() throws Exception { // ensure that the headers from the incoming request are used instead those of the stored watch // otherwise the watch would run as the user who stored the watch, but it needs to be run as the user who // executes this request @@ -141,8 +148,6 @@ public class TransportExecuteWatchAction extends WatcherTransportAction + { + "watch" : { + "trigger": { + "schedule": { + "interval": "1d" + } + }, + "input": { + "simple": { + "foo": "bar" + } + }, + "actions": { + "my-logging": { + "transform": { + "script": { + "source": "def x = [:] ; def y = [:] ; x.a = y ; y.a = x ; return x" + } + }, + "logging": { + "text": "{{ctx}}" + } + } + } + } + } + + - match: { watch_record.watch_id: "_inlined_" } + - match: { watch_record.trigger_event.type: "manual" } + - match: { watch_record.state: "executed" } + - match: { watch_record.result.actions.0.status: "failure" } + - match: { watch_record.result.actions.0.error.caused_by.caused_by.type: "illegal_argument_exception" } + - match: { watch_record.result.actions.0.error.caused_by.caused_by.reason: "Iterable object is self-referencing itself" } + + - do: + catch: bad_request + xpack.watcher.execute_watch: + body: > + { + "watch": { + "trigger": { + "schedule": { + "interval": "10s" + } + }, + "input": { + "simple": { + "foo": "bar" + } + }, + "actions": { + "my-logging": { + "transform": { + "script": { + "source": "def x = [:] ; def y = [:] ; x.a = y ; y.a = x ; return x" + } + }, + "logging": { + "text": "{{#join}}ctx.payload{{/join}}" + } + } + } + } + } From 223d3c1f4c1c35e30658d93e5140698f4a19eb48 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Tue, 23 Jan 2018 08:41:48 -0800 Subject: [PATCH 10/13] [DOCS] Added QA notes about scheduled events (elastic/x-pack-elasticsearch#3641) Original commit: elastic/x-pack-elasticsearch@64b67aa0ad2a0a460e58cd719a062c77181604e8 --- docs/en/ml/calendars.asciidoc | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/docs/en/ml/calendars.asciidoc b/docs/en/ml/calendars.asciidoc index 4e3ba10c6d7..34bd3f10a19 100644 --- a/docs/en/ml/calendars.asciidoc +++ b/docs/en/ml/calendars.asciidoc @@ -9,10 +9,15 @@ The {ml} model is not ill-affected and you do not receive spurious results. You can create calendars and scheduled events in the **Settings** pane on the **Machine Learning** page in {kib} or by using {ref}/ml-apis.html[{ml} APIs]. -A scheduled event must have a start time, end time, and description. You can -identify zero or more scheduled events in a calendar. Jobs can then subscribe to -calendars and the {ml} analytics handle all subsequent scheduled events -appropriately. +A scheduled event must have a start time, end time, and description. In general, +scheduled events are short in duration (typically lasting from a few hours to a +day) and occur infrequently. If you have regularly occurring events, such as +weekly maintenance periods, you do not need to create scheduled events for these +circumstances; they are already handled by the {ml} analytics. + +You can identify zero or more scheduled events in a calendar. Jobs can then +subscribe to calendars and the {ml} analytics handle all subsequent scheduled +events appropriately. If you want to add multiple scheduled events at once, you can import an iCalendar (`.ics`) file in {kib} or a JSON file in the @@ -21,8 +26,16 @@ add events to calendar API //] . -NOTE: Bucket results are generated during scheduled events but they have an +[NOTE] +-- + +* If your iCalendar file contains recurring events, only the first occurrence is +imported. +* Bucket results are generated during scheduled events but they have an anomaly score of zero. For more information about bucket results, see {ref}/ml-results-resource.html[Results Resources]. +* If you use long or frequent scheduled events, it might take longer for the +{ml} analytics to learn to model your data and some anomalous behavior might be +missed. -//TO-DO: Add screenshot showing special events in Single Metric Viewer? +-- From 697a08e742bb28f39c4ec428a599f420cff3a9b6 Mon Sep 17 00:00:00 2001 From: David Roberts Date: Tue, 23 Jan 2018 16:55:08 +0000 Subject: [PATCH 11/13] [ML] Refactor so ML doesn't require PersistentTasksService at startup (elastic/x-pack-elasticsearch#3682) At present the PersistentTasksService is created inside the ML plugin. This is undesirable, as other plugins will use persistent tasks in the near future. This change refactors the startup code so that the PersistentTasksService no longer needs to be passed to any constructors for ML components. A future change will still be required to actually move the initialization of the PersistentTasksClusterService, PersistentTasksService and PersistentTasksExecutorRegistry out of the ML plugin, but following this change it should be fairly simple. Original commit: elastic/x-pack-elasticsearch@3c2a8e020e31960437f7bdf5555d6da33cce96c2 --- .../persistent/AllocatedPersistentTask.java | 11 ++++++++++ .../xpack/ml/MachineLearning.java | 20 +++++++++---------- .../xpack/ml/datafeed/DatafeedManager.java | 20 ++++++++----------- .../ml/datafeed/DatafeedManagerTests.java | 6 +----- 4 files changed, 30 insertions(+), 27 deletions(-) diff --git a/plugin/core/src/main/java/org/elasticsearch/xpack/persistent/AllocatedPersistentTask.java b/plugin/core/src/main/java/org/elasticsearch/xpack/persistent/AllocatedPersistentTask.java index 353f7ab9495..87949e16913 100644 --- a/plugin/core/src/main/java/org/elasticsearch/xpack/persistent/AllocatedPersistentTask.java +++ b/plugin/core/src/main/java/org/elasticsearch/xpack/persistent/AllocatedPersistentTask.java @@ -11,6 +11,7 @@ import org.apache.logging.log4j.util.Supplier; import org.elasticsearch.action.ActionListener; import org.elasticsearch.action.admin.cluster.node.tasks.cancel.CancelTasksRequest; import org.elasticsearch.common.Nullable; +import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.tasks.CancellableTask; import org.elasticsearch.tasks.Task; import org.elasticsearch.tasks.TaskCancelledException; @@ -19,6 +20,7 @@ import org.elasticsearch.tasks.TaskManager; import java.util.Map; import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Predicate; /** * Represents a executor node operation that corresponds to a persistent task @@ -105,6 +107,15 @@ public class AllocatedPersistentTask extends CancellableTask { COMPLETED // the task is done running and trying to notify caller } + /** + * Waits for this persistent task to have the desired state. + */ + public void waitForPersistentTaskStatus(Predicate> predicate, + @Nullable TimeValue timeout, + PersistentTasksService.WaitForPersistentTaskStatusListener listener) { + persistentTasksService.waitForPersistentTaskStatus(persistentTaskId, predicate, timeout, listener); + } + public void markAsCompleted() { completeAndNotifyIfNeeded(null); } diff --git a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/MachineLearning.java b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/MachineLearning.java index 691aef78fe8..0b02a042d41 100644 --- a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/MachineLearning.java +++ b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/MachineLearning.java @@ -360,8 +360,7 @@ public class MachineLearning extends Plugin implements ActionPlugin, AnalysisPlu PersistentTasksService persistentTasksService = new PersistentTasksService(settings, clusterService, threadPool, client); - components.addAll(createComponents(client, clusterService, threadPool, xContentRegistry, environment, resourceWatcherService, - persistentTasksService)); + components.addAll(createComponents(client, clusterService, threadPool, xContentRegistry, environment)); // This was lifted from the XPackPlugins createComponents when it got split // This is not extensible and anyone copying this code needs to instead make this work @@ -379,10 +378,11 @@ public class MachineLearning extends Plugin implements ActionPlugin, AnalysisPlu return components; } - public Collection createComponents(Client client, ClusterService clusterService, ThreadPool threadPool, - NamedXContentRegistry xContentRegistry, Environment environment, - ResourceWatcherService resourceWatcherService, - PersistentTasksService persistentTasksService) { + // TODO: once initialization of the PersistentTasksClusterService, PersistentTasksService + // and PersistentTasksExecutorRegistry has been moved somewhere else the entire contents of + // this method can replace the entire contents of the overridden createComponents() method + private Collection createComponents(Client client, ClusterService clusterService, ThreadPool threadPool, + NamedXContentRegistry xContentRegistry, Environment environment) { if (enabled == false || transportClientMode || tribeNode || tribeNodeClient) { return emptyList(); } @@ -426,12 +426,13 @@ public class MachineLearning extends Plugin implements ActionPlugin, AnalysisPlu this.autodetectProcessManager.set(autodetectProcessManager); DatafeedJobBuilder datafeedJobBuilder = new DatafeedJobBuilder(client, jobProvider, auditor, System::currentTimeMillis); DatafeedManager datafeedManager = new DatafeedManager(threadPool, client, clusterService, datafeedJobBuilder, - System::currentTimeMillis, auditor, persistentTasksService); + System::currentTimeMillis, auditor); this.datafeedManager.set(datafeedManager); MlLifeCycleService mlLifeCycleService = new MlLifeCycleService(environment, clusterService, datafeedManager, autodetectProcessManager); - InvalidLicenseEnforcer invalidLicenseEnforcer = - new InvalidLicenseEnforcer(settings, getLicenseState(), threadPool, datafeedManager, autodetectProcessManager); + + // This object's constructor attaches to the license state, so there's no need to retain another reference to it + new InvalidLicenseEnforcer(settings, getLicenseState(), threadPool, datafeedManager, autodetectProcessManager); return Arrays.asList( mlLifeCycleService, @@ -442,7 +443,6 @@ public class MachineLearning extends Plugin implements ActionPlugin, AnalysisPlu jobDataCountsPersister, datafeedManager, auditor, - invalidLicenseEnforcer, new MlAssignmentNotifier(settings, auditor, clusterService) ); } diff --git a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/DatafeedManager.java b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/DatafeedManager.java index d0a843ad41d..a917d2fa618 100644 --- a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/DatafeedManager.java +++ b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/DatafeedManager.java @@ -32,7 +32,6 @@ import org.elasticsearch.xpack.ml.job.messages.Messages; import org.elasticsearch.xpack.ml.notifications.Auditor; import org.elasticsearch.xpack.persistent.PersistentTasksCustomMetaData; import org.elasticsearch.xpack.persistent.PersistentTasksCustomMetaData.PersistentTask; -import org.elasticsearch.xpack.persistent.PersistentTasksService; import java.util.ArrayList; import java.util.Iterator; @@ -55,7 +54,6 @@ public class DatafeedManager extends AbstractComponent { private final Client client; private final ClusterService clusterService; - private final PersistentTasksService persistentTasksService; private final ThreadPool threadPool; private final Supplier currentTimeSupplier; private final Auditor auditor; @@ -66,14 +64,13 @@ public class DatafeedManager extends AbstractComponent { private volatile boolean isolated; public DatafeedManager(ThreadPool threadPool, Client client, ClusterService clusterService, DatafeedJobBuilder datafeedJobBuilder, - Supplier currentTimeSupplier, Auditor auditor, PersistentTasksService persistentTasksService) { + Supplier currentTimeSupplier, Auditor auditor) { super(Settings.EMPTY); this.client = Objects.requireNonNull(client); this.clusterService = Objects.requireNonNull(clusterService); this.threadPool = threadPool; this.currentTimeSupplier = Objects.requireNonNull(currentTimeSupplier); this.auditor = Objects.requireNonNull(auditor); - this.persistentTasksService = Objects.requireNonNull(persistentTasksService); this.datafeedJobBuilder = Objects.requireNonNull(datafeedJobBuilder); clusterService.addListener(taskRunner); } @@ -91,8 +88,7 @@ public class DatafeedManager extends AbstractComponent { ActionListener datafeedJobHandler = ActionListener.wrap( datafeedJob -> { - Holder holder = new Holder(task.getPersistentTaskId(), task.getAllocationId(), datafeed, datafeedJob, - task.isLookbackOnly(), new ProblemTracker(auditor, job.getId()), taskHandler); + Holder holder = new Holder(task, datafeed, datafeedJob, new ProblemTracker(auditor, job.getId()), taskHandler); runningDatafeedsOnThisNode.put(task.getAllocationId(), holder); task.updatePersistentStatus(DatafeedState.STARTED, new ActionListener>() { @Override @@ -279,7 +275,7 @@ public class DatafeedManager extends AbstractComponent { public class Holder { - private final String taskId; + private final TransportStartDatafeedAction.DatafeedTask task; private final long allocationId; private final DatafeedConfig datafeed; // To ensure that we wait until loopback / realtime search has completed before we stop the datafeed @@ -291,13 +287,13 @@ public class DatafeedManager extends AbstractComponent { volatile Future future; private volatile boolean isRelocating; - Holder(String taskId, long allocationId, DatafeedConfig datafeed, DatafeedJob datafeedJob, boolean autoCloseJob, + Holder(TransportStartDatafeedAction.DatafeedTask task, DatafeedConfig datafeed, DatafeedJob datafeedJob, ProblemTracker problemTracker, Consumer handler) { - this.taskId = taskId; - this.allocationId = allocationId; + this.task = task; + this.allocationId = task.getAllocationId(); this.datafeed = datafeed; this.datafeedJob = datafeedJob; - this.autoCloseJob = autoCloseJob; + this.autoCloseJob = task.isLookbackOnly(); this.problemTracker = problemTracker; this.handler = handler; } @@ -397,7 +393,7 @@ public class DatafeedManager extends AbstractComponent { return; } - persistentTasksService.waitForPersistentTaskStatus(taskId, Objects::isNull, TimeValue.timeValueSeconds(20), + task.waitForPersistentTaskStatus(Objects::isNull, TimeValue.timeValueSeconds(20), new WaitForPersistentTaskStatusListener() { @Override public void onResponse(PersistentTask persistentTask) { diff --git a/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/datafeed/DatafeedManagerTests.java b/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/datafeed/DatafeedManagerTests.java index 8322d951fa8..b3122d599ac 100644 --- a/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/datafeed/DatafeedManagerTests.java +++ b/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/datafeed/DatafeedManagerTests.java @@ -41,7 +41,6 @@ import org.elasticsearch.xpack.ml.notifications.Auditor; import org.elasticsearch.xpack.ml.notifications.AuditorField; import org.elasticsearch.xpack.persistent.PersistentTasksCustomMetaData; import org.elasticsearch.xpack.persistent.PersistentTasksCustomMetaData.PersistentTask; -import org.elasticsearch.xpack.persistent.PersistentTasksService; import org.junit.Before; import org.mockito.ArgumentCaptor; @@ -122,8 +121,6 @@ public class DatafeedManagerTests extends ESTestCase { when(threadPool.executor(MachineLearning.DATAFEED_THREAD_POOL_NAME)).thenReturn(executorService); when(threadPool.executor(ThreadPool.Names.GENERIC)).thenReturn(executorService); - PersistentTasksService persistentTasksService = mock(PersistentTasksService.class); - datafeedJob = mock(DatafeedJob.class); when(datafeedJob.isRunning()).thenReturn(true); when(datafeedJob.stop()).thenReturn(true); @@ -135,8 +132,7 @@ public class DatafeedManagerTests extends ESTestCase { return null; }).when(datafeedJobBuilder).build(any(), any(), any()); - datafeedManager = new DatafeedManager(threadPool, client, clusterService, datafeedJobBuilder, () -> currentTime, auditor, - persistentTasksService); + datafeedManager = new DatafeedManager(threadPool, client, clusterService, datafeedJobBuilder, () -> currentTime, auditor); verify(clusterService).addListener(capturedClusterStateListener.capture()); } From d065b087ee1bde075132346d46712249a96073cc Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Tue, 23 Jan 2018 18:08:54 +0100 Subject: [PATCH 12/13] Revert "Cleanup: Remove HaltedClock (elastic/x-pack-elasticsearch#3664)" This reverts commit elastic/x-pack-elasticsearch@f91c401a60c28d2c22bf1a3c5c206988cf2d9e1b due to failing tests, like ./gradlew :x-pack-elasticsearch:plugin:watcher:test -Dtests.seed=AE30350FCE96D26D -Dtests.class=org.elasticsearch.xpack.watcher.watch.WatchTests -Dtests.method="testParserSelfGenerated" -Dtests.security.manager=true -Dtests.locale=ja-JP -Dtests.timezone=EET Original commit: elastic/x-pack-elasticsearch@e45d79d6430cbaf4bcae5e90746ccd7469d91d6e --- .../xpack/common/time/HaltedClock.java | 47 +++++++++++++++++++ .../SamlLogoutRequestMessageBuilderTests.java | 11 +++-- .../xpack/watcher/watch/WatchParser.java | 8 ++-- 3 files changed, 56 insertions(+), 10 deletions(-) create mode 100644 plugin/core/src/main/java/org/elasticsearch/xpack/common/time/HaltedClock.java diff --git a/plugin/core/src/main/java/org/elasticsearch/xpack/common/time/HaltedClock.java b/plugin/core/src/main/java/org/elasticsearch/xpack/common/time/HaltedClock.java new file mode 100644 index 00000000000..809a737abb6 --- /dev/null +++ b/plugin/core/src/main/java/org/elasticsearch/xpack/common/time/HaltedClock.java @@ -0,0 +1,47 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.xpack.common.time; + +import org.joda.time.DateTime; +import org.joda.time.DateTimeZone; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneId; +import java.time.ZoneOffset; + +public class HaltedClock extends Clock { + + private final DateTime now; + + public HaltedClock(DateTime now) { + this.now = now.toDateTime(DateTimeZone.UTC); + } + + @Override + public ZoneId getZone() { + return ZoneOffset.UTC; + } + + @Override + public Clock withZone(ZoneId zoneId) { + if (zoneId.equals(ZoneOffset.UTC)) { + return this; + } + + throw new IllegalArgumentException("Halted clock time zone cannot be changed"); + } + + @Override + public long millis() { + return now.getMillis(); + } + + @Override + public Instant instant() { + return Instant.ofEpochMilli(now.getMillis()); + } +} diff --git a/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlLogoutRequestMessageBuilderTests.java b/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlLogoutRequestMessageBuilderTests.java index 252cf2f0d1c..88d574de489 100644 --- a/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlLogoutRequestMessageBuilderTests.java +++ b/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlLogoutRequestMessageBuilderTests.java @@ -6,10 +6,11 @@ package org.elasticsearch.xpack.security.authc.saml; import java.time.Clock; -import java.time.Instant; -import java.time.ZoneOffset; +import org.elasticsearch.xpack.common.time.HaltedClock; import org.hamcrest.Matchers; +import org.joda.time.DateTime; +import org.joda.time.DateTimeZone; import org.junit.Before; import org.opensaml.saml.common.xml.SAMLConstants; import org.opensaml.saml.saml2.core.LogoutRequest; @@ -66,8 +67,8 @@ public class SamlLogoutRequestMessageBuilderTests extends SamlTestCase { "http://idp.example.com/saml/logout/artifact"); idpRole.getSingleLogoutServices().add(sloArtifact); - Clock fixedClock = Clock.fixed(Instant.now(), ZoneOffset.UTC); - final SamlLogoutRequestMessageBuilder builder = new SamlLogoutRequestMessageBuilder(fixedClock, sp, idp, nameId, session); + final DateTime now = DateTime.now(DateTimeZone.UTC); + final SamlLogoutRequestMessageBuilder builder = new SamlLogoutRequestMessageBuilder(new HaltedClock(now), sp, idp, nameId, session); final LogoutRequest logoutRequest = builder.build(); assertThat(logoutRequest, notNullValue()); assertThat(logoutRequest.getReason(), nullValue()); @@ -81,7 +82,7 @@ public class SamlLogoutRequestMessageBuilderTests extends SamlTestCase { assertThat(logoutRequest.getConsent(), nullValue()); assertThat(logoutRequest.getNotOnOrAfter(), nullValue()); assertThat(logoutRequest.getIssueInstant(), notNullValue()); - assertThat(logoutRequest.getIssueInstant().getMillis(), equalTo(fixedClock.millis())); + assertThat(logoutRequest.getIssueInstant(), equalTo(now)); assertThat(logoutRequest.getSessionIndexes(), iterableWithSize(1)); assertThat(logoutRequest.getSessionIndexes().get(0).getSessionIndex(), equalTo(session)); assertThat(logoutRequest.getDestination(), equalTo("http://idp.example.com/saml/logout/redirect")); diff --git a/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java b/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java index bb68d1217ae..7a8dda3fc86 100644 --- a/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java +++ b/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchParser.java @@ -29,12 +29,11 @@ import org.elasticsearch.xpack.watcher.support.xcontent.WatcherXContentParser; import org.elasticsearch.xpack.watcher.transform.ExecutableTransform; import org.elasticsearch.xpack.watcher.trigger.Trigger; import org.elasticsearch.xpack.watcher.trigger.TriggerService; +import org.elasticsearch.xpack.common.time.HaltedClock; import org.joda.time.DateTime; import java.io.IOException; import java.time.Clock; -import java.time.Instant; -import java.time.ZoneOffset; import java.util.Collections; import java.util.HashMap; import java.util.List; @@ -103,9 +102,8 @@ public class WatchParser extends AbstractComponent { XContentParser parser = null; try { // EMPTY is safe here because we never use namedObject - Clock fixedClock = Clock.fixed(Instant.now(), ZoneOffset.UTC); - parser = new WatcherXContentParser(xContentType.xContent().createParser(NamedXContentRegistry.EMPTY, source), fixedClock, - withSecrets ? cryptoService : null); + parser = new WatcherXContentParser(xContentType.xContent().createParser(NamedXContentRegistry.EMPTY, source), + new HaltedClock(now), withSecrets ? cryptoService : null); parser.nextToken(); return parse(id, includeStatus, parser); } catch (IOException ioe) { From 97e018f1bd287a1cb8b89381f2a974ac2284c75f Mon Sep 17 00:00:00 2001 From: David Kyle Date: Tue, 23 Jan 2018 17:40:41 +0000 Subject: [PATCH 13/13] [ML] Return the updated calendar after removing a job (elastic/x-pack-elasticsearch#3690) Original commit: elastic/x-pack-elasticsearch@1bf1f3228bbc6d79dacd15258a1316fff69b826b --- .../xpack/ml/rest/calendar/RestDeleteCalendarJobAction.java | 4 ++-- .../test/resources/rest-api-spec/test/ml/calendar_crud.yml | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/rest/calendar/RestDeleteCalendarJobAction.java b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/rest/calendar/RestDeleteCalendarJobAction.java index 749987755b9..54dd6885018 100644 --- a/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/rest/calendar/RestDeleteCalendarJobAction.java +++ b/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/rest/calendar/RestDeleteCalendarJobAction.java @@ -10,7 +10,7 @@ import org.elasticsearch.common.settings.Settings; import org.elasticsearch.rest.BaseRestHandler; import org.elasticsearch.rest.RestController; import org.elasticsearch.rest.RestRequest; -import org.elasticsearch.rest.action.AcknowledgedRestListener; +import org.elasticsearch.rest.action.RestToXContentListener; import org.elasticsearch.xpack.ml.MachineLearning; import org.elasticsearch.xpack.ml.action.UpdateCalendarJobAction; import org.elasticsearch.xpack.ml.calendars.Calendar; @@ -39,6 +39,6 @@ public class RestDeleteCalendarJobAction extends BaseRestHandler { String jobId = restRequest.param(Job.ID.getPreferredName()); UpdateCalendarJobAction.Request request = new UpdateCalendarJobAction.Request(calendarId, Collections.emptySet(), Collections.singleton(jobId)); - return channel -> client.execute(UpdateCalendarJobAction.INSTANCE, request, new AcknowledgedRestListener<>(channel)); + return channel -> client.execute(UpdateCalendarJobAction.INSTANCE, request, new RestToXContentListener<>(channel)); } } diff --git a/plugin/src/test/resources/rest-api-spec/test/ml/calendar_crud.yml b/plugin/src/test/resources/rest-api-spec/test/ml/calendar_crud.yml index fa16bb43bf3..7a78aa6cb51 100644 --- a/plugin/src/test/resources/rest-api-spec/test/ml/calendar_crud.yml +++ b/plugin/src/test/resources/rest-api-spec/test/ml/calendar_crud.yml @@ -237,6 +237,8 @@ xpack.ml.delete_calendar_job: calendar_id: "wildlife" job_id: "tiger" + - match: { calendar_id: "wildlife" } + - length: { job_ids: 0 } - do: xpack.ml.get_calendars: