* master:
Security: revert to old way of merging automata (#32254)
Networking: Fix test leaking buffer (#32296)
Undo a debugging change that snuck in during the field aliases merge.
Painless: Update More Methods to New Naming Scheme (#32305)
[TEST] Fix assumeFalse -> assumeTrue in SSLReloadIntegTests
Ingest: Support integer and long hex values in convert (#32213)
Introduce fips_mode setting and associated checks (#32326)
Add V_6_3_3 version constant
[DOCS] Removed extraneous callout number.
Rest HL client: Add put license action (#32214)
Add ERR to ranking evaluation documentation (#32314)
Introduce Application Privileges with support for Kibana RBAC (#32309)
Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
[Kerberos] Add Kerberos authentication support (#32263)
[ML] Extract persistent task methods from MlMetadata (#32319)
Add Restore Snapshot High Level REST API
Register ERR metric with NamedXContentRegistry (#32320)
fixes broken build for third-party-tests (#32315)
Allow Integ Tests to run in a FIPS-140 JVM (#31989)
[DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280)
awaitsfix testRandomClusterStateUpdates
[TEST] add version skip to weighted_avg tests
Consistent encoder names (#29492)
Add WeightedAvg metric aggregation (#31037)
Switch monitoring to new style Requests (#32255)
Rename ranking evaluation `quality_level` to `metric_score` (#32168)
Fix a test bug around nested aggregations and field aliases. (#32287)
Add new permission for JDK11 to load JAAS libraries (#32132)
Silence SSL reload test that fails on JDK 11
[test] package pre-install java check (#32259)
specify subdirs of lib, bin, modules in package (#32253)
Switch x-pack:core to new style Requests (#32252)
awaitsfix SSLConfigurationReloaderTests
Painless: Clean up add methods in PainlessLookup (#32258)
Fail shard if IndexShard#storeStats runs into an IOException (#32241)
AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated
Remove unnecessary warning supressions (#32250)
CCE when re-throwing "shard not available" exception in TransportShardMultiGetAction (#32185)
Add new fields to monitoring template for Beats state (#32085)
This commit reverts to the pre-6.3 way of merging automata as the
change in 6.3 significantly impacts the performance for roles with a
large number of concrete indices. In addition, the maximum number of
states for security automata has been increased to 100,000 in order
to allow users to use roles that caused problems pre-6.3 and 6.3 fixed.
As an escape hatch, the maximum number of states is configurable with
a setting so that users with complex patterns in roles can increase
the states with the knowledge that there is more memory usage.
In the HL REST client we replace the License object with a string, because of
complexity of this class. It is also not really needed on the client side since
end-users are not interacting with the license besides passing it as a string
to the server.
Relates #29827
This commit introduces "Application Privileges" to the X-Pack security
model.
Application Privileges are managed within Elasticsearch, and can be
tested with the _has_privileges API, but do not grant access to any
actions or resources within Elasticsearch. Their purpose is to allow
applications outside of Elasticsearch to represent and store their own
privileges model within Elasticsearch roles.
Access to manage application privileges is handled in a new way that
grants permission to specific application names only. This lays the
foundation for more OLS on cluster privileges, which is implemented by
allowing a cluster permission to inspect not just the action being
executed, but also the request to which the action is applied.
To support this, a "conditional cluster privilege" is introduced, which
is like the existing cluster privilege, except that it has a Predicate
over the request as well as over the action name.
Specifically, this adds
- GET/PUT/DELETE actions for defining application level privileges
- application privileges in role definitions
- application privileges in the has_privileges API
- changes to the cluster permission class to support checking of request
objects
- a new "global" element on role definition to provide cluster object
level security (only for manage application privileges)
- changes to `kibana_user`, `kibana_dashboard_only_user` and
`kibana_system` roles to use and manage application privileges
Closes#29820Closes#31559
This bundles the x-pack:protocol project into the x-pack:plugin:core
project because we'd like folks to consider it an implementation detail
of our build rather than a separate artifact to be managed and depended
on. It is now bundled into both x-pack:plugin:core and
client:rest-high-level. To make this work I had to fix a few things.
Firstly, I had to make PluginBuildPlugin work with the shadow plugin.
In that case we have to bundle only the `shadow` dependencies and the
shadow jar.
Secondly, every reference to x-pack:plugin:core has to use the `shadow`
configuration. Without that the reference is missing all of the
un-shadowed dependencies. I tried to make it so that applying the shadow
plugin automatically redefines the `default` configuration to mirror the
`shadow` configuration which would allow us to use bare project references
to the x-pack:plugin:core project but I couldn't make it work. It'd *look*
like it works but then fail for transitive dependencies anyway. I think
it is still a good thing to do but I don't have the willpower to do it
now.
Finally, I had to fix an issue where Eclipse and IntelliJ didn't properly
reference shadowed transitive dependencies. Neither IDE supports shadowing
natively so they have to reference the shadowed projects. We fix this by
detecting `shadow` dependencies when in "Intellij mode" or "Eclipse mode"
and adding `runtime` dependencies to the same target. This convinces
IntelliJ and Eclipse to play nice.
This commit adds support for Kerberos authentication with a platinum
license. Kerberos authentication support relies on SPNEGO, which is
triggered by challenging clients with a 401 response with the
`WWW-Authenticate: Negotiate` header. A SPNEGO client will then provide
a Kerberos ticket in the `Authorization` header. The tickets are
validated using Java's built-in GSS support. The JVM uses a vm wide
configuration for Kerberos, so there can be only one Kerberos realm.
This is enforced by a bootstrap check that also enforces the existence
of the keytab file.
In many cases a fallback authentication mechanism is needed when SPNEGO
authentication is not available. In order to support this, the
DefaultAuthenticationFailureHandler now takes a list of failure response
headers. For example, one realm can provide a
`WWW-Authenticate: Negotiate` header as its default and another could
provide `WWW-Authenticate: Basic` to indicate to the client that basic
authentication can be used in place of SPNEGO.
In order to test Kerberos, unit tests are run against an in-memory KDC
that is backed by an in-memory ldap server. A QA project has also been
added to test against an actual KDC, which is provided by the krb5kdc
fixture.
Closes#30243
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:core` project to use the new versions.
New data is reported from Beats to the monitoring endpoint. This PR adds the template change necessary for it. See https://github.com/elastic/beats/issues/7521 for more details.
Queue data is skipped for now as implementation is not finished yet.
* es/master: (23 commits)
Switch full-cluster-restart to new style Requests (#32140)
[DOCS] Clarified that you must remove X-Pack plugin when upgrading from pre-6.3. (#32016)
Remove BouncyCastle dependency from runtime (#32193)
INGEST: Extend KV Processor (#31789) (#32232)
INGEST: Make a few Processors callable by Painless (#32170)
Add region ISO code to GeoIP Ingest plugin (#31669)
[Tests] Remove QueryStringQueryBuilderTests#toQuery class assertions (#32236)
Make sure that field aliases count towards the total fields limit. (#32222)
Switch rolling restart to new style Requests (#32147)
muting failing test for internal auto date histogram to avoid failure before fix is merged
MINOR: Remove unused `IndexDynamicSettings` (#32237)
Fix multi level nested sort (#32204)
Enhance Parent circuit breaker error message (#32056)
[ML] Use default request durability for .ml-state index (#32233)
Remove indices stats timeout from monitoring docs
Rename ranking evaluation response section (#32166)
Dependencies: Upgrade to joda time 2.10 (#32160)
Remove aliases resolution limitations when security is enabled (#31952)
Ensure that field aliases cannot be used in multi-fields. (#32219)
TESTS: Check for Netty resource leaks (#31861)
...
* Remove BouncyCastle dependency from runtime
This commit introduces a new gradle project that contains
the classes that have a dependency on BouncyCastle. For
the default distribution, It builds a jar from those and
in puts it in a subdirectory of lib
(/tools/security-cli) along with the BouncyCastle jars.
This directory is then passed in the
ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools
that use these classes.
BouncyCastle is removed as a runtime dependency (remains
as a compileOnly one) from x-pack core and x-pack security.
* master:
Painless: Simplify Naming in Lookup Package (#32177)
Handle missing values in painless (#32207)
add support for write index resolution when creating/updating documents (#31520)
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
Remove indication of future multi-homing support (#32187)
Rest test - allow for snapshots to take 0 milliseconds
Make x-pack-core generate a pom file
Rest HL client: Add put watch action (#32026)
Build: Remove pom generation for plugin zip files (#32180)
Fix comments causing errors with Java 11
Fix rollup on date fields that don't support epoch_millis (#31890)
Detect and prevent configuration that triggers a Gradle bug (#31912)
[test] port linux package packaging tests (#31943)
Revert "Introduce a Hashing Processor (#31087)" (#32178)
Remove empty @return from JavaDoc
Adjust SSLDriver behavior for JDK11 changes (#32145)
[test] use randomized runner in packaging tests (#32109)
Add support for field aliases. (#32172)
Painless: Fix caching bug and clean up addPainlessClass. (#32142)
Call setReferences() on custom referring tokenfilters in _analyze (#32157)
Fix BwC Tests looking for UUID Pre 6.4 (#32158)
Improve docs for search preferences (#32159)
use before instead of onOrBefore
Add more contexts to painless execute api (#30511)
Add EC2 credential test for repository-s3 (#31918)
A replica can be promoted and started in one cluster state update (#32042)
Fix Java 11 javadoc compile problem
Fix CP for namingConventions when gradle home has spaces (#31914)
Fix `range` queries on `_type` field for singe type indices (#31756)
[DOCS] Update TLS on Docker for 6.3 (#32114)
ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are
Remove versionType from translog (#31945)
Switch distribution to new style Requests (#30595)
Build: Skip jar tests if jar disabled
Painless: Add PainlessClassBuilder (#32141)
Build: Make additional test deps of check (#32015)
Disable C2 from using AVX-512 on JDK 10 (#32138)
Build: Move shadow customizations into common code (#32014)
Painless: Fix Bug with Duplicate PainlessClasses (#32110)
Remove empty @param from Javadoc
Re-disable packaging tests on suse boxes
Docs: Fix missing example script quote (#32010)
[ML] Wait for aliases in multi-node tests (#32086)
[ML] Move analyzer dependencies out of categorization config (#32123)
Ensure to release translog snapshot in primary-replica resync (#32045)
Handle TokenizerFactory TODOs (#32063)
Relax TermVectors API to work with textual fields other than TextFieldType (#31915)
Updates the build to gradle 4.9 (#32087)
Mute :qa:mixed-cluster indices.stats/10_index/Index - all’
Check that client methods match API defined in the REST spec (#31825)
Enable testing in FIPS140 JVM (#31666)
Fix put mappings java API documentation (#31955)
Add exclusion option to `keep_types` token filter (#32012)
[Test] Modify assert statement for ssl handshake (#32072)
Prior to 6.3 a trial license default to security enabled. Since 6.3
they default to security disabled. If a cluster is upgraded from <6.3
to >6.3, then we detect this and mimic the old behaviour with respect
to security.
Relates #29827
This implementation behaves like the current transport client, that you basically cannot configure a Watch POJO representation as an argument to the put watch API, but only a bytes reference. You can use the the `WatchSourceBuilder` from the `org.elasticsearch.plugin:x-pack-core` dependency to build watches.
This commit also changes the license type to trial, so that watcher is available in high level rest client tests.
/cc @hub-cap
The ML config classes will shortly be moved to the X-Pack protocol
library to allow the ML APIs to be moved to the high level REST
client. Dependencies on server functionality should be removed
from the config classes before this is done.
This change is entirely about moving code between packages. It
does not add or remove any functionality or tests.
Ensure our tests can run in a FIPS JVM
JKS keystores cannot be used in a FIPS JVM as attempting to use one
in order to init a KeyManagerFactory or a TrustManagerFactory is not
allowed.( JKS keystore algorithms for private key encryption are not
FIPS 140 approved)
This commit replaces JKS keystores in our tests with the
corresponding PEM encoded key and certificates both for key and trust
configurations.
Whenever it's not possible to refactor the test, i.e. when we are
testing that we can load a JKS keystore, etc. we attempt to
mute the test when we are running in FIPS 140 JVM. Testing for the
JVM is naive and is based on the name of the security provider as
we would control the testing infrastrtucture and so this would be
reliable enough.
Other cases of tests being muted are the ones that involve custom
TrustStoreManagers or KeyStoreManagers, null TLS Ciphers and the
SAMLAuthneticator class as we cannot sign XML documents in the
way we were doing. SAMLAuthenticator tests in a FIPS JVM can be
reenabled with precomputed and signed SAML messages at a later stage.
IT will be covered in a subsequent PR
* es/master:
Add Index UUID to `/_stats` Response (#31871)
Painless: Move and Rename Several Methods in the lookup package (#32105)
Bypass highlight query terms extraction on empty fields (#32090)
Switch non-x-pack to new style requests (#32106)
[Rollup] Add new capabilities endpoint for concrete rollup indices (#30401)
Revert "[test] disable packaging tests for suse boxes"
SQL: allow LEFT and RIGHT as function names (#32066)
DOCS: put LIMIT 10 to the SQL query (#32065)
[test] turn on host io cache for opensuse (#32053)
Tweaked Elasticsearch Service links for SEO
This introduces a new GetRollupIndexCaps API which allows the user to retrieve rollup capabilities of a specific rollup index (or index pattern). This is distinct from the existing RollupCaps endpoint.
- Multiple jobs can be stored in multiple indices and point to a single target data index pattern (logstash-*). The existing API finds capabilities/config of all jobs matching that data index pattern.
- One rollup index can hold data from multiple jobs, targeting multiple data index patterns. This new API finds the capabilities based on the concrete rollup indices.
* es/master: (21 commits)
Tweaked Elasticsearch Service links for SEO
Watcher: Store username on watch execution (#31873)
Use correct formatting for links (#29460)
Painless: Separate PainlessLookup into PainlessLookup and PainlessLookupBuilder (#32054)
Scripting: Remove dead code from painless module (#32064)
[Rollup] Replace RollupIT with a ESRestTestCase version (#31977)
[TEST] Consistent algorithm usage (#32077)
[Rollup] Fix duplicate field names in test (#32075)
Ensure only parent breaker trips in unit test
Unmute field collapsing rest tests
Fix BWC check after backport
[Tests] Fix failure due to changes exception message (#32036)
Remove unused params from SSource and Walker (#31935)
[Test] Mute MlJobIT#testDeleteJobAfterMissingAliases
Turn off real-mem breaker in REST tests
Turn off real-mem breaker in single node tests
Fix broken OpenLDAP Vagrant QA test
Cleanup Duplication in `PainlessScriptEngine` (#31991)
SCRIPTING: Remove unused MultiSearchTemplateRequestBuilder (#32049)
Fix compile issues introduced by merge (#32058)
...
There is currently no way to see what user executed a watch. This commit
adds the decrypted username to each execution in the watch history, in a
new field "user".
Closes#31772
Metric config already whitelist scaled_floats, but it wasn't added to
the histo group config. This centralizes the mapping types map
so that both metrics and histo (and any future configs) use the same
map.
Fixes#32035
This commit adds the _xpack/usage api to the high level rest client.
Currently in the transport api, the usage data is exposed in a limited
fashion, at most giving one level of helper methods for the inner keys
of data, but then exposing thos subobjects as maps of objects. Rather
than making parsers for every set of usage data from each feature, this
PR exposes the entire set of usage data as a map of maps.
Previously, the ensureWatchExists was overridable. This commit makes
it final so that it cannot be overridden, and cleans up some redundant
code in the process.
We can leverage the composite agg's new `missing_bucket` feature on
terms groupings. This means the aggregation criteria used in the indexer
will now return null buckets for missing keys.
Because all buckets are now returned (even if a key is null),
we can guarantee correct doc counts with
"combined" jobs (where a job rolls up multiple schemas). This was
previously impossible since composite would ignore documents that
didn't have _all_ the keys, meaning non-overlapping schemas would
cause composite to return no buckets.
Note: date_histo does not use `missing_bucket`, since a timestamp is
always required.
The docs have been adjusted to recommend a single, combined job. It
also makes reference to the previous issue to help users that are upgrading
(rather than just deleting the sections).
Historically we have loaded SSL objects (such as SSLContext,
SSLIOSessionStrategy) by passing in the SSL settings, constructing a
new SSL configuration from those settings and then looking for a
cached object that matches those settings.
The primary issue with this approach is that it requires a fully
configured Settings object to be available any time the SSL context
needs to be loaded. If the Settings include SecureSettings (such as
passwords for keys or keystores) then this is not true, and the cached
SSL object cannot be loaded at runtime.
This commit introduces an alternative approach of naming every cached
ssl configuration, so that it is possible to load the SSL context for
a named configuration (such as "xpack.http.ssl"). This means that the
calling code does not need to have ongoing access to the secure
settings that were used to load the configuration.
This change also allows monitoring exporters to use SSL passwords
from secure settings, however an exporter that uses a secure SSL setting
(e.g. truststore.secure_password) may not have its SSL settings updated
dynamically (this is prevented by a settings validator).
Exporters without secure settings can continue to be defined and updated
dynamically.
* master:
[TEST] Mute SlackMessageTests.testTemplateRender
Docs: Explain closing the high level client
[ML] Re-enable memory limit integration tests (#31328)
[test] disable packaging tests for suse boxes
Add nio transport to security plugin (#31942)
XContentTests : Insert random fields at random positions (#30867)
Force execution of fetch tasks (#31974)
Fix unreachable error condition in AmazonS3Fixture (#32005)
Tests: Fix SearchFieldsIT.testDocValueFields (#31995)
Add Expected Reciprocal Rank metric (#31891)
[ML] Get ForecastRequestStats doc in RestoreModelSnapshotIT (#31973)
SQL: Add support for single parameter text manipulating functions (#31874)
[ML] Ensure immutability of MlMetadata (#31957)
Tests: Mute SearchFieldsIT.testDocValueFields()
muted tests due to #31940
Work around reported problem in eclipse (#31960)
Move build integration tests out of :buildSrc project (#31961)
Tests: Remove use of joda time in some tests (#31922)
[Test] Reactive 3rd party tests on CI (#31919)
SQL: Support for escape sequences (#31884)
SQL: HAVING clause should accept only aggregates (#31872)
Docs: fix typo in datehistogram (#31972)
Switch url repository rest tests to new style requests (#31944)
Switch reindex tests to new style requests (#31941)
Docs: Added note about cloud service to installation and getting started
[DOCS] Removes alternative docker pull example (#31934)
Add Snapshots Status API to High Level Rest Client (#31515)
ingest: date_index_name processor template resolution (#31841)
Test: fix null failure in watcher test (#31968)
Switch test framework to new style requests (#31939)
Switch low level rest tests to new style Requests (#31938)
Switch high level rest tests to new style requests (#31937)
[ML] Mute test failing due to Java 11 date time format parsing bug (#31899)
[TEST] Mute SlackMessageTests.testTemplateRender
Fix assertIngestDocument wrongfully passing (#31913)
Remove unused reference to filePermissionsCache (#31923)
rolling upgrade should use a replica to prevent relocations while running a scroll
HLREST: Bundle the x-pack protocol project (#31904)
Increase logging level for testStressMaybeFlush
Added lenient flag for synonym token filter (#31484)
[X-Pack] Beats centralized management: security role + licensing (#30520)
HLRest: Move xPackInfo() to xPack().info() (#31905)
Docs: add security delete role to api call table (#31907)
[test] port archive distribution packaging tests (#31314)
Watcher: Slack message empty text (#31596)
[ML] Mute failing DetectionRulesIT.testCondition() test
Fix broken NaN check in MovingFunctions#stdDev() (#31888)
Date: Add DateFormatters class that uses java.time (#31856)
[ML] Switch native QA tests to a 3 node cluster (#31757)
Change trappy float comparison (#31889)
Fix building AD URL from domain name (#31849)
Add opaque_id to audit logging (#31878)
re-enable backcompat tests
add support for is_write_index in put-alias body parsing (#31674)
Improve release notes script (#31833)
[DOCS] Fix broken link in painless example
Handle missing values in painless (#30975)
Remove the ability to index or query context suggestions without context (#31007)
Ingest: Enable Templated Fieldnames in Rename (#31690)
[Docs] Fix typo in the Rollup API Quick Reference (#31855)
Ingest: Add ignore_missing option to RemoveProc (#31693)
Add template config for Beat state to X-Pack Monitoring (#31809)
Watcher: Add ssl.trust email account setting (#31684)
Remove link to oss-MSI (#31844)
Painless: Restructure Definition/Whitelist (#31879)
HLREST: Add x-pack-info API (#31870)
This is related to #27260. It adds the SecurityNioTransport to the
security plugin. Additionally, it adds support for ip filtering. And it
randomly uses the nio transport in security integration tests.
The test failure in #31916 revealed that updating
rules on a job was modifying the detectors list
in-place. That meant the old cluster state and the
updated cluster state had no difference and thus the
change was not propagated to non-master nodes.
This commit fixes that and also reviews all of ML
metadata in order to ensure immutability.
Closes#31916
Currently Role.Builder keeps a reference to the FieldPermissionsCache that is
passed into its constructors. This seems to be unused except for passing it on
to convertFromIndicesPrivileges() in the second ctor itself, but we don't need
to keep the internal reference in that case, so it can be removed.
Relates to #31876
* Adding Beats x-pack plugin + index templates
* Adding built-in roles for Beats central management
* Fixing typo
* Refactoring: extract common code into method
* More refactoring for more code reuse
* Use a single index for Beats management
* Rename "fragment" to "block"
* Adding configuration block type
* Expand kibana_system role to include Beats management index privileges
* Fixing syntax
* Adding test
* Adding asserting for reserved role
* Fixing privileges
* Updating template
* Removing beats plugin
* Fixing tests
* Fixing role variable name
* Fixing assertions
* Switching to preferred syntax for boolean false checks
* Making class final
* Making variables final
* Updating Basic license message to be more accurate
Originally I put the X-Pack info object into the top level rest client
object. I did that because we thought we'd like to squash `xpack` from
the name of the X-Pack APIs now that it is part of the default
distribution. We still kind of want to do that, but at least for now we
feel like it is better to keep the high level rest client aligned with
the other language clients like C# and Python. This shifts the X-Pack
info API to align with its json spec file.
Relates to #31870
With https://github.com/elastic/beats/pull/7075 Beats introduces state reporting for X-Pack Monitoring. The data sent up to Elasticsearch ends up stored in the following format.
```
"beats_state": {
"timestamp": "2018-07-05T07:21:03.581Z",
"state": {
"module": {
"count": 1,
"names": [
"http"
]
}
},
"beat": {
"uuid": "594039b5-6353-4d78-9bad-778ecc0fe83f",
"type": "metricbeat",
"version": "7.0.0-alpha1",
"name": "ruflin",
"host": "ruflin"
}
}
```
This PR adds the new fields to the template.
This is the first x-pack API we're adding to the high level REST client
so there is a lot to talk about here!
= Open source
The *client* for these APIs is open source. We're taking the previously
Elastic licensed files used for the `Request` and `Response` objects and
relicensing them under the Apache 2 license.
The implementation of these features is staying under the Elastic
license. This lines up with how the rest of the Elasticsearch language
clients work.
= Location of the new files
We're moving all of the `Request` and `Response` objects that we're
relicensing to the `x-pack/protocol` directory. We're adding a copy of
the Apache 2 license to the root fo the `x-pack/protocol` directory to
line up with the language in the root `LICENSE.txt` file. All files in
this directory will have the Apache 2 license header as well. We don't
want there to be any confusion. Even though the files are under the
`x-pack` directory, they are Apache 2 licensed.
We chose this particular directory layout because it keeps the X-Pack
stuff together and easier to think about.
= Location of the API in the REST client
We've been following the layout of the rest-api-spec files for other
APIs and we plan to do this for the X-Pack APIs with one exception:
we're dropping the `xpack` from the name of most of the APIs. So
`xpack.graph.explore` will become `graph().explore()` and
`xpack.license.get` will become `license().get()`.
`xpack.info` and `xpack.usage` are special here though because they
don't belong to any proper category. For now I'm just calling
`xpack.info` `xPackInfo()` and intend to call usage `xPackUsage` though
I'm not convinced that this is the final name for them. But it does get
us started.
= Jars, jars everywhere!
This change makes the `xpack:protocol` project a `compile` scoped
dependency of the `x-pack:plugin:core` and `client:rest-high-level`
projects. I intend to keep it a compile scoped dependency of
`x-pack:plugin:core` but I intend to bundle the contents of the protocol
jar into the `client:rest-high-level` jar in a follow up. This change
has grown large enough at this point.
In that followup I'll address javadoc issues as well.
= Breaking-Java
This breaks that transport client by a few classes around. We've
traditionally been ok with doing this to the transport client.
* master:
REST high-level client: add get index API (#31703)
SQL: Allow long literals (#31777)
SQL: Fix incorrect message for aliases (#31792)
Test: Do not remove xpack templates when cleaning (#31642)
Reduce more raw types warnings (#31780)
Add unreleased version 6.3.2
Scripting: Remove support for deprecated StoredScript contexts (#31394)
[ML][TEST] Use java 11 valid time format in DataDescriptionTests (#31817)
[ML] Don't treat stale FAILED jobs as OPENING in job allocation (#31800)
[ML] Fix calendar and filter updates from non-master nodes (#31804)
Fix license header generation on Windows (#31790)
mark RollupIT.testTwoJobsStartStopDeleteOne as AwaitsFix
mark SearchAsyncActionTests.testFanOutAndCollect as AwaitsFix
Correct exclusion of test on JDK 11
Fix doclint jdk 11
Add JDK11 support and enable in CI (#31644)
Watcher: Fix check for currently executed watches (#31137)
Watcher: Ensure correct method is used to read secure settings (#31753)
SQL: Update CLI logo
It seems that java 11 tightened some validations with regard to
time formats. The random instance creator was setting an odd
time format to the data description which is invalid when run
with java 11. This commit changes it to a valid format.
* Upgrade bouncycastle
Required to fix
`bcprov-jdk15on-1.55.jar; invalid manifest format `
on jdk 11
* Downgrade bouncycastle to avoid invalid manifest
* Add checksum for new jars
* Update tika permissions for jdk 11
* Mute test failing on jdk 11
* Add JDK11 to CI
* Thread#stop(Throwable) was removed
http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html
* Disable failing tests #31456
* Temprorarily disable doc tests
To see if there are other failures on JDK11
* Only blacklist specific doc tests
* Disable only failing tests in ingest attachment plugin
* Mute failing HDFS tests #31498
* Mute failing lang-painless tests #31500
* Fix backwards compatability builds
Fix JAVA version to 10 for ES 6.3
* Add 6.x to bwx -> java10
* Prefix out and err from buildBwcVersion for readability
```
> Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion
[bwc] :buildSrc:compileJava
[bwc] WARNING: An illegal reflective access operation has occurred
[bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize()
[bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
[bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[bwc] WARNING: All illegal access operations will be denied in a future release
[bwc] :buildSrc:compileGroovy
[bwc] :buildSrc:writeVersionProperties
[bwc] :buildSrc:processResources
[bwc] :buildSrc:classes
[bwc] :buildSrc:jar
```
* Also set RUNTIME_JAVA_HOME for bwcBuild
So that we can make sure it's not too new for the build to understand.
* Align bouncycastle dependency
* fix painles array tets
closes#31500
* Update jar checksums
* Keep 8/10 runtime/compile untill consensus builds on 11
* Only skip failing tests if running on Java 11
* Failures are dependent of compile java version not runtime
* Condition doc test exceptions on compiler java version as well
* Disable hdfs tests based on runtime java
* Set runtime java to minimum supported for bwc
* PR review
* Add comment with ticket for forbidden apis
* master:
[ML] Rate limit established model memory updates (#31768)
[Docs] Correct default window_size (#31582)
S3 fixture should report 404 on unknown bucket (#31782)
Detach Transport from TransportService (#31727)
[ML] Limit ML filter items to 10K (#31731)
[ML] Return statistics about forecasts as part of the jobsstats and usage API (#31647)
Fixture for Minio testing (#31688)
[DOCS] Add missing get mappings docs to HLRC (#31765)
[DOCS] Starting Elasticsearch (#31701)
Painless: Complete Removal of Painless Type (#31699)
Fix not waiting for Netty ThreadDeathWatcher in IT (#31758)
Consolidate watcher setting update registration (#31762)
Build: re-enabled bwc (#31769)
ingest: Introduction of a bytes processor (#31733)
Fix coerce validation_method in GeoBoundingBoxQueryBuilder (#31747)
Add analyze API to high-level rest client (#31577)
[DOCS] Typos
DOC: Add examples to the SQL docs (#31633)
Add support for AWS session tokens (#30414)
Watcher: Reenable start/stop yaml tests (#31754)
Implemented XContent serialisation for GetIndexResponse (#31675)
JDBC: Fix stackoverflow on getObject and timestamp conversion (#31735)
resolveHasher defaults to NOOP (#31723)
Account for XContent overhead in in-flight breaker
Split CircuitBreaker-related tests (#31659)
Add write*Blob option to replace existing blob (#31729)
Painless: Add Context Docs (#31190)
Watcher: Fix chain input toXcontent serialization (#31721)
Docs: Match the examples in the description (#31710)
rest-high-level: added get cluster settings (#31706)
[Docs] Correct typos (#31720)
Clean up double semicolon code typos (#31687)
[DOCS] Check for Windows and *nix file paths (#31648)
[ML] Validate ML filter_id (#31535)
Revert long lines
Fix TransportChangePasswordActionTests
Today TransportService is tightly coupled with Transport since it
requires an instance of TransportService in order to receive responses
and send requests. This is mainly due to the Request and Response handlers
being maintained in TransportService but also because of the lack of a proper
callback interface.
This change moves request handler registry and response handler registration into
Transport and adds all necessary methods to `TransportConnectionListener` in order
to remove the `TransportService` dependency from `Transport`
Transport now accepts one or more `TransportConnectionListener` instances that are
executed sequentially in a blocking fashion.
Add hard limit to the number of items
a filter may have. This serves to protect
from excessive overhead due to the filters
taking too much memory or lookups becoming
too expensive.
This change adds stats about forecasts, to the jobstats api as well as xpack/_usage. The following
information is collected:
_xpack/ml/anomaly_detectors/{jobid|_all}/_stats:
- total number of forecasts
- memory statistics (mean/min/max)
- runtime statistics
- record statistics
- counts by status
_xpack/usage
- collected by job status as well as overall (_all):
- total number of forecasts
- number of jobs that have at least 1 forecast
- memory, runtime, record statistics
- counts by status
Fixes#31395
* Default resolveFromHash to Hasher.NOOP
This changes the default behavior when resolving the hashing
algorithm from unrecognised hash strings, which was introduced in
#31234
A hash string that doesn't start with an algorithm identifier can
either be a malformed/corrupted hash or a plaintext password when
Hasher.NOOP is used(against warnings).
Do not make assumptions about which of the two is true for such
strings and default to Hasher.NOOP. Hash verification will subsequently
fail for malformed hashes.
Finally, do not log the potentially malformed hash as this can very
well be a plaintext password.
Resolves#31697
Reverts 58cf95a06f
* master:
Do not check for object existence when deleting repository index files (#31680)
Remove extra check for object existence in repository-gcs read object (#31661)
Support multiple system store types (#31650)
[Test] Clean up some repository-s3 tests (#31601)
[Docs] Use capital letters in section headings (#31678)
[DOCS] Add PQL language Plugin (#31237)
Merge AzureStorageService and AzureStorageServiceImpl and clean up tests (#31607)
TEST: Fix test task invocation (#31657)
Revert "[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg"
Fix RealmInteg test failures
Extend allowed characters for grok field names (#21745) (#31653)
[DOCS] Fix licensing API details (#31667)
[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg
Fix CreateSnapshotRequestTests Failure (#31630)
Configurable password hashing algorithm/cost (#31234)
[TEST] Mute failing NamingConventionsTaskIT tests
[DOCS] Replace CONFIG_DIR with ES_PATH_CONF (#31635)
Core: Require all actions have a Task (#31627)
Nhat Nguyen
Jay Modi
Igor Motov
Tim Vernum
Nik Everett
Yogesh Gaikwad
David Kyle
Andy Bristol
Christoph Büscher
Nicolas Ruflin
Martijn van Groningen
Ioannis Kakavas
Ryan Ernst
Alexander Reelsen
David Roberts
Zachary Tong
Michael Basnight
Jim Ferenczi
Tim Brooks
Dimitris Athanasiou
Shaunak Kashyap
Alpar Torok
Simon Willnauer
Hendrik Muhs