Commit Graph

40175 Commits

Author SHA1 Message Date
Nik Everett 11a2189c6f
Switch x-pack full restart to new style Requests (#32294)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:qa:full-cluster-restart` project to use
the new versions.
2018-07-26 12:11:31 -04:00
Tim Brooks 7a56df7c98
Release requests in cors handler (#32364)
There are two scenarios where a http request could terminate in the cors
handler. If that occurs, the requests need to be released. This commit
releases those requests.
2018-07-26 10:06:24 -06:00
Jack Conradson df579f8bce
Painless: Clean Up PainlessClass Variables (#32380)
Removes the variables name, clazz, and type as they are unnecessary. Renames 
staticMembers -> staticFields, members -> fields, getters -> getterMethodHandles, and 
setters -> setterMethodHandles.
2018-07-26 09:02:06 -07:00
Igor Motov 3b050194f9
Docs: Fix callouts in put license HL REST docs (#32363) 2018-07-26 09:00:35 -07:00
David Roberts 0afa265ac9
[ML] Consistent pattern for strict/lenient parser names (#32399)
Previously we had two patterns for naming of strict
and lenient parsers.

Some classes had CONFIG_PARSER and METADATA_PARSER,
and used an enum to pass the parser type to nested
parsers.

Other classes had STRICT_PARSER and LENIENT_PARSER
and used ternary operators to pass the parser type
to nested parsers.

This change makes all ML classes use the second of
the patterns described above.
2018-07-26 16:55:40 +01:00
mzbeck 63a0436764 Update update-settings.asciidoc (#31378) 2018-07-26 17:16:53 +02:00
Christoph Büscher 35ae87125d
Remove some dead code (#31993)
Removing some dead code or supressing warnings where apropriate. Most of the
time the variable tested for null is dereferenced earlier or never used before.
2018-07-26 17:12:51 +02:00
Jason Tedor eb675a1c4d
Introduce index store plugins (#32375)
Today we allow plugins to add index store implementations yet we are not
doing this in our new way of managing plugins as pull versus push. That
is, today we still allow plugins to push index store providers via an on
index module call where they can turn around and add an index
store. Aside from being inconsistent with how we manage plugins today
where we would look to pull such implementations from plugins at node
creation time, it also means that we do not know at a top-level (for
example, in the indices service) which index stores are available. This
commit addresses this by adding a dedicated plugin type for index store
plugins, removing the index module hook for adding index stores, and by
aggregating these into the top-level of the indices service.
2018-07-26 08:05:49 -04:00
Christoph Büscher bec888fa78 Rank-Eval: Reduce scope of an unchecked supression
We should only supress the unchecked warnings on ConstructingObjectParser.
2018-07-26 11:16:01 +02:00
Adrien Grand f7ba14d5f4
Make sure _forcemerge respects `max_num_segments`. (#32291)
An upcoming [Lucene change](https://issues.apache.org/jira/browse/LUCENE-7976)
will make TieredMergePolicy respect the maximum merged segment size all the
time, meaning it will possibly not respect the `max_num_segments` parameter
anymore if the shard is larger than the maximum segment size.

This change makes sure that `max_num_segments` is respected for now in order
to give us time to think about how to integrate this change, and also to delay
it until 7.0 as this might be a big-enough change for us to wait for a new
major version.
2018-07-26 09:02:58 +02:00
Armin Braun 48885d2b7e
TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests (#32377)
* TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests

* Release all ref counted things that weren't getting properly released
* Mannually force channel promise to be completed because mock channel doesn't do it and it prevents one `release` call in `io.netty.channel.ChannelOutboundHandlerAdapter#write` from firing
2018-07-26 07:01:59 +02:00
Jason Tedor 467a60ba0c
Only enforce password hashing check if FIPS enabled (#32383)
This commit modifies the FIPS password hashing algorithm check to only
be executed if FIPS mode is enabled.
2018-07-25 20:57:03 -04:00
lcawl a577fb3381 [DOCS] Fix formatting error in Slack action 2018-07-25 14:19:19 -07:00
Jack Conradson 10bfedeb53
Painless: Fix documentation links to use existing refs (#32335)
Uses ref and xpack-ref instead of custom ones in the Painless docs for
long-term maintainability.
2018-07-25 10:59:25 -07:00
Jack Conradson 853aa0afb4
Painless: Decouple PainlessLookupBuilder and Whitelists (#32346)
Implements a static function in PainlessLookupBuilder that contains all the logic related
to Whitelist.  PainlessLookupBuilder is available for use in loading from methods beyond
Whitelist now.
2018-07-25 10:52:01 -07:00
Lisa Cawley 8067250fbe
[DOCS] Adds recommendation for xpack.security.enabled (#32345) 2018-07-25 09:51:40 -07:00
Dimitris Athanasiou de53f0123f [TEST] Mute ConvertProcessortTests.testConvertIntHexError
Relates #32370
2018-07-25 17:35:23 +01:00
Zachary Tong 6cf7588c3d
[TEST] Fix failure due to exception message in java11 (#32321)
Java 11 uses more verbose exceptions messages, causing this assertion
to fail.  Changed the test to be less restrictive and only look
for the classes we care about.
2018-07-25 11:34:26 -04:00
lcawl 5a12c63b4e [DOCS] Fixes typo in ML aggregations page 2018-07-25 08:32:01 -07:00
lcawl 6832aa6797 [DOCS] Adds link from bucket_span property to common time units 2018-07-25 08:31:52 -07:00
Dimitris Athanasiou 9a7a649755
[ML][DOCS] Add documentation for detector rules and filters (#32013) 2018-07-25 16:10:32 +01:00
Igor Motov e12e2e0cdd
Add opaque_id to index audit logging (#32260)
Logs opaque_id if it is available with all audit log messages using
index-based audit log.

Closes #31521
2018-07-25 08:07:09 -07:00
Dimitris Athanasiou 9efd1407d5 Add 6.5.0 version to master
This commit adds the 6.5.0 version constant to the master branch.
2018-07-25 15:57:48 +01:00
Vladimir Dolzhenko a2547168d2
fixes broken build for third-party-tests (#32353)
fixes broken build repository-s3 for third-party-tests
2018-07-25 16:34:30 +02:00
Alpar Torok 6aea82956c
Number of utilities for writing gradle integration tests (#32282)
These are collected from a number of open PRs and are required to
improove existing and write more readable future tests.
I am extracting them to their own PR hoping to be able to merge and use
them sooner.
2018-07-25 06:22:54 +00:00
Alpar Torok b6c14935d7
Determine the minimum gradle version based on the wrapper (#32226)
* Determine the minimum gradle version based on the wrapper

This is restrictive and forces users of the plugin to move together with
us, but without integration tests it's close to impossible to make sure
that the claimed compatability is really there.
If we do want to offer more flexibility, we should add those tests
first.

* Track gradle version in individual file

* PR review
2018-07-25 06:21:03 +00:00
Ioannis Kakavas d0fd23669a
Enable FIPS JVM in CI (#32330)
Now that #31666 and #31989 are merged we can run our tests in
fips JVM. This commits enables us to run tests on a Java 8
JVM using BouncyCastleFIPS as a security Provider.
2018-07-25 08:48:19 +03:00
Tim Vernum 33e3822c5d Fix JarHell on X-Pack protocol
Ports 466bc8c from 6.x
2018-07-25 14:14:19 +10:00
Jay Modi e43375bf9a
Security: revert to old way of merging automata (#32254)
This commit reverts to the pre-6.3 way of merging automata as the
change in 6.3 significantly impacts the performance for roles with a
large number of concrete indices. In addition, the maximum number of
states for security automata has been increased to 100,000 in order
to allow users to use roles that caused problems pre-6.3 and 6.3 fixed.

As an escape hatch, the maximum number of states is configurable with
a setting so that users with complex patterns in roles can increase
the states with the knowledge that there is more memory usage.
2018-07-24 16:26:50 -06:00
Armin Braun 717df26fc3
Networking: Fix test leaking buffer (#32296)
* Test `handler` must release buffer the same way the replaced `org.elasticsearch.http.netty4.Netty4HttpRequestHandler#channelRead0` releases it
* Closes #32289
2018-07-24 23:04:22 +02:00
Julie Tibshirani 54ba3ea286 Undo a debugging change that snuck in during the field aliases merge. 2018-07-24 13:19:37 -07:00
Jack Conradson 1690451a9f
Painless: Update More Methods to New Naming Scheme (#32305)
This finishes the updating the methods in the PainlessLookupBuilder to the new naming scheme. Mechanical change. Methods include the ones used for copying members in the inheritance hierarchy, calculating shortcuts, and setting the functional interface.
2018-07-24 13:08:05 -07:00
Lee Hinman 73be9840dd [TEST] Fix assumeFalse -> assumeTrue in SSLReloadIntegTests 2018-07-24 13:10:55 -06:00
Ryan Ernst 49d4b26f16
Ingest: Support integer and long hex values in convert (#32213)
This commit adds checks for hex formatted strings in the convert
processor, allowing strings like `0x1` to be parsed as integer `1`.

closes #32182
2018-07-24 12:05:50 -07:00
Ioannis Kakavas be40a6982a
Introduce fips_mode setting and associated checks (#32326)
* Introduce fips_mode setting and associated checks

Introduce xpack.security.fips_mode.enabled setting ( default false)
When it is set to true, a number of Bootstrap checks are performed:
- Check that Secure Settings are of the latest version (3)
- Check that no JKS keystores are configured
- Check that compliant algorithms ( PBKDF2 family ) are used for
  password hashing
2018-07-24 22:05:00 +03:00
Lee Hinman e0b7e4b1c5 Add V_6_3_3 version constant
Relates to #32329
2018-07-24 12:56:43 -06:00
debadair 32eb4d41b6
[DOCS] Removed extraneous callout number. 2018-07-24 11:51:52 -07:00
Igor Motov 29c802f88e
Rest HL client: Add put license action (#32214)
In the HL REST client we replace the License object with a string, because of 
complexity of this class. It is also not really needed on the client side since 
end-users are not interacting with the license besides passing it as a string 
to the server.

Relates #29827
2018-07-24 11:19:07 -07:00
Christoph Büscher c1cc0cef61
Add ERR to ranking evaluation documentation (#32314)
This change adds a section about the Expected Reciprocal Rank metric (ERR) to
the Ranking Evaluation documentation.
2018-07-24 19:58:34 +02:00
Tim Vernum 387c3c7f1d Introduce Application Privileges with support for Kibana RBAC (#32309)
This commit introduces "Application Privileges" to the X-Pack security
model.

Application Privileges are managed within Elasticsearch, and can be
tested with the _has_privileges API, but do not grant access to any
actions or resources within Elasticsearch. Their purpose is to allow
applications outside of Elasticsearch to represent and store their own
privileges model within Elasticsearch roles.

Access to manage application privileges is handled in a new way that
grants permission to specific application names only. This lays the
foundation for more OLS on cluster privileges, which is implemented by
allowing a cluster permission to inspect not just the action being
executed, but also the request to which the action is applied.
To support this, a "conditional cluster privilege" is introduced, which
is like the existing cluster privilege, except that it has a Predicate
over the request as well as over the action name.

Specifically, this adds
- GET/PUT/DELETE actions for defining application level privileges
- application privileges in role definitions
- application privileges in the has_privileges API
- changes to the cluster permission class to support checking of request
  objects
- a new "global" element on role definition to provide cluster object
  level security (only for manage application privileges)
- changes to `kibana_user`, `kibana_dashboard_only_user` and
  `kibana_system` roles to use and manage application privileges

Closes #29820
Closes #31559
2018-07-24 10:34:46 -06:00
Nik Everett e6b9f59e4e
Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
This bundles the x-pack:protocol project into the x-pack:plugin:core
project because we'd like folks to consider it an implementation detail
of our build rather than a separate artifact to be managed and depended
on. It is now bundled into both x-pack:plugin:core and
client:rest-high-level. To make this work I had to fix a few things.

Firstly, I had to make PluginBuildPlugin work with the shadow plugin.
In that case we have to bundle only the `shadow` dependencies and the
shadow jar.

Secondly, every reference to x-pack:plugin:core has to use the `shadow`
configuration. Without that the reference is missing all of the
un-shadowed dependencies. I tried to make it so that applying the shadow
plugin automatically redefines the `default` configuration to mirror the
`shadow` configuration which would allow us to use bare project references
to the x-pack:plugin:core project but I couldn't make it work. It'd *look*
like it works but then fail for transitive dependencies anyway. I think
it is still a good thing to do but I don't have the willpower to do it
now.

Finally, I had to fix an issue where Eclipse and IntelliJ didn't properly
reference shadowed transitive dependencies. Neither IDE supports shadowing
natively so they have to reference the shadowed projects. We fix this by
detecting `shadow` dependencies when in "Intellij mode" or "Eclipse mode"
and adding `runtime` dependencies to the same target. This convinces
IntelliJ and Eclipse to play nice.
2018-07-24 11:53:04 -04:00
Yogesh Gaikwad a525c36c60 [Kerberos] Add Kerberos authentication support (#32263)
This commit adds support for Kerberos authentication with a platinum
license. Kerberos authentication support relies on SPNEGO, which is
triggered by challenging clients with a 401 response with the
`WWW-Authenticate: Negotiate` header. A SPNEGO client will then provide
a Kerberos ticket in the `Authorization` header. The tickets are
validated using Java's built-in GSS support. The JVM uses a vm wide
configuration for Kerberos, so there can be only one Kerberos realm.
This is enforced by a bootstrap check that also enforces the existence
of the keytab file.

In many cases a fallback authentication mechanism is needed when SPNEGO
authentication is not available. In order to support this, the
DefaultAuthenticationFailureHandler now takes a list of failure response
headers. For example, one realm can provide a
`WWW-Authenticate: Negotiate` header as its default and another could
provide `WWW-Authenticate: Basic` to indicate to the client that basic
authentication can be used in place of SPNEGO.

In order to test Kerberos, unit tests are run against an in-memory KDC
that is backed by an in-memory ldap server. A QA project has also been
added to test against an actual KDC, which is provided by the krb5kdc
fixture.

Closes #30243
2018-07-24 08:44:26 -06:00
David Kyle 99426eb4f8
[ML] Extract persistent task methods from MlMetadata (#32319)
Move ML persistent task helper functions to the new class MlTasks
and remove MLMetadataField after moving the string constant to 
MlMetadata.
2018-07-24 15:22:57 +01:00
Daniel Mitterdorfer 73a38895fd
Add Restore Snapshot High Level REST API
With this commit we add the restore snapshot API to the Java high level
REST client.

Relates #27205
Relates #32155
2018-07-24 16:17:09 +02:00
Christoph Büscher 59cf600e03
Register ERR metric with NamedXContentRegistry (#32320)
This adds the ERR metric to the provided xContent parsers in the module and the
high level rest client registry. Also adding integration tests to make sure the
metric is correctly registered and usable from the client.
2018-07-24 16:05:43 +02:00
Vladimir Dolzhenko 46709f1406
fixes broken build for third-party-tests (#32315)
fixes broken build for third-party-tests (#32315)
Relates #31918 / Closes infra/issues/6085
2018-07-24 14:09:41 +02:00
Ioannis Kakavas a2dbd83db1
Allow Integ Tests to run in a FIPS-140 JVM (#31989)
* Complete changes for running IT in a fips JVM

- Mute :x-pack:qa:sql:security:ssl:integTest as it
  cannot run in FIPS 140 JVM until the SQL CLI supports key/cert.
- Set default JVM keystore/truststore password in top level build
  script for all integTest tasks in a FIPS 140 JVM
- Changed top level x-pack build script to use keys and certificates
  for trust/key material when spinning up clusters for IT
2018-07-24 12:48:14 +03:00
David Kyle 177750719d [DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280) 2018-07-24 10:23:13 +01:00
Andy Bristol d07b4ecfa3 awaitsfix testRandomClusterStateUpdates
For #32308
2018-07-23 17:20:01 -07:00
Zachary Tong 7c43da280b [TEST] add version skip to weighted_avg tests
Added in 6.4, so we don't want this running on pre-6.4 BWC tests
2018-07-23 19:50:26 -04:00