518 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe
ce38127a16
Merge branch 'master' into index-lifecycle 2018-06-14 13:02:24 +01:00
Luca Cavanna
ce245a7320
Remove RestGetAllAliasesAction (#31308)
We currently have a specific REST action to retrieve all aliaes, which
uses internally the get index API. This doesn't seem to be required
anymore though as the existing RestGetAliaesAction could as well take
the requests with no indices and aliases specified.

This commit removes the RestGetAllAliasesAction in favour of using
RestGetAliasesAction also for requests that don't specify indices nor
aliases. Similar to #31129.
2018-06-14 11:21:16 +02:00
Tim Vernum
89a24698b1 Temporary fix for broken build
x-pack/sql depends on lang-painless which depends on ASM 5.1
FeatureAwareCheck needs ASM 6
This is a hack to strip ASM5 from the classpath for FeatureAwareCheck
2018-06-14 18:22:24 +10:00
Tanguy Leroux
4d7447cb5e
Reenable Checkstyle's unused import rule (#31270) 2018-06-14 09:52:46 +02:00
Tanguy Leroux
2d4c9ce08c Remove remaining unused imports before merging #31270 2018-06-14 09:52:03 +02:00
Tim Vernum
74c6f18fea Fix non-REST doc snippet 2018-06-14 16:26:17 +10:00
Tal Levy
b9e1a905eb fix retryrequest test 2018-06-13 16:04:54 -07:00
Tal Levy
11ef3b90a3 Merge branch 'master' into index-lifecycle 2018-06-13 15:56:47 -07:00
Costin Leau
870a913217 [DOC] Extend SQL docs
Add overview section
Add data type section
Improve function section
2018-06-14 00:38:53 +03:00
Tim Brooks
700357d04e
Immediately flush channel after writing to buffer (#31301)
This is related to #27260. Currently when we queue a write with a
channel we set OP_WRITE and wait until the next selection loop to flush
the write. However, if the channel does not have a pending write, it
is probably ready to flush. This PR implements an optimistic flush logic
that will attempt this flush.
2018-06-13 15:32:13 -06:00
Tal Levy
04e0d0a5e7 Merge branch 'master' into index-lifecycle 2018-06-13 14:15:28 -07:00
lcawl
509729f9c1 [DOCS] Shortens ML API intros 2018-06-13 13:43:15 -07:00
Tal Levy
bbb7889236
move security ingest processors to a sub ingest directory (#31306)
It makes sense to introduce new Security ingest
processors (example: #31087), and this change would
give them a good place to be written.
2018-06-13 13:35:41 -07:00
Costin Leau
43cb24035e
SQL: Whitelist SQL utility class for better scripting (#30681)
Add SQL class for reusing code inside SQL functions within Painless

Fix #29832
2018-06-13 23:08:18 +03:00
Zachary Tong
d4262de83a
[Docs] All Rollup docs experimental, agg limitations, clarify DeleteJob (#31299)
- All rollup pages should be marked as experimental instead of just
the top page
- While the job config docs state which aggregations are allowed, adding
a section which specifically details this in one place is more convenient
for the user
- Add a clarification that the DeleteJob API does not delete the rollup
data, just the rollup job.
2018-06-13 15:42:20 -04:00
Colin Goodheart-Smithe
6c9e0aeca3
Adds API to assign or change the policy for an index (#31277)
* Adds API to assign or change the policy for an index

This api will change `index.lifecycle.name` for all indexes in the
provided pattern as long as they are not currently in the shrink action.

Later changes will loosen this restriction so an index is only reject
if it is currently in the shrink action AND the diff between the old
and new policy includes changes to the shrink action.

Also later changes will detect if the current step is not present in
the new policy and move the index onto the next available step

* Changes name to SetPolicyForIndexAction

Also changes all related Classes and api endpoints

* fixes tests
2018-06-13 17:17:32 +01:00
Colin Goodheart-Smithe
6785391a5f
Merge branch 'master' into index-lifecycle 2018-06-13 16:38:35 +01:00
Zachary Tong
a486177a19
[Rollup] Metric config parser must use builder so validation runs (#31159)
The parser for the Metric config was directly instantiating
the config object, rather than using the builder.  That means it was
bypassing the validation logic built into the builder, and would allow
users to create invalid metric configs (like using unsupported metrics).

The job would later blow up and abort due to bad configs, but this isn't
immediately obvious to the user since the PutJob API succeeded.
2018-06-13 11:31:04 -04:00
Colin Goodheart-Smithe
43eaec304b
Fixes ExplainLifecycleRequestTests 2018-06-13 16:15:26 +01:00
David Kyle
88f44a9f66
[ML] Check licence when datafeeds use cross cluster search (#31247)
This change prevents a datafeed using cross cluster search from starting if the remote cluster
does not have x-pack installed and a sufficient license. The check is made only when starting a 
datafeed.
2018-06-13 15:42:18 +01:00
Colin Goodheart-Smithe
9fd4f3f854
Merge branch 'master' into index-lifecycle 2018-06-13 13:41:17 +01:00
Tom Veasey
66f7dd2c4d
[ML] Update test thresholds to account for changes to memory control (#31289)
To avoid temporary failures, this also disables these tests until elastic/ml-cpp#122 is committed.
2018-06-13 13:12:53 +01:00
Dimitris Athanasiou
5c77ebe89d
[ML] Implement new rules design (#31110)
Rules allow users to supply a detector with domain
knowledge that can improve the quality of the results.
The model detects statistically anomalous results but it
has no knowledge of the meaning of the values being modelled.

For example, a detector that performs a population analysis
over IP addresses could benefit from a list of IP addresses
that the user knows to be safe. Then anomalous results for
those IP addresses will not be created and will not affect
the quantiles either.

Another example would be a detector looking for anomalies
in the median value of CPU utilization. A user might want
to inform the detector that any results where the actual
value is less than 5 is not interesting.

This commit introduces a `custom_rules` field to the `Detector`.
A detector may have multiple rules which are combined with `or`.

A rule has 3 fields: `actions`, `scope` and `conditions`.

Actions is a list of what should happen when the rule applies.
The current options include `skip_result` and `skip_model_update`.
The default value for `actions` is the `skip_result` action.

Scope is optional and allows for applying filters on any of the
partition/over/by field. When not defined the rule applies to
all series. The `filter_id` needs to be specified to match the id
of the filter to be used. Optionally, the `filter_type` can be specified
as either `include` (default) or `exclude`. When set to `include`
the rule applies to entities that are in the filter. When set to
`exclude` the rule only applies to entities not in the filter.

There may be zero or more conditions. A condition requires `applies_to`,
`operator` and `value` to be specified. The `applies_to` value can be
either `actual`, `typical` or `diff_from_typical` and it specifies
the numerical value to which the condition applies. The `operator`
(`lt`, `lte`, `gt`, `gte`) and `value` complete the definition.
Conditions are combined with `and` and allow to specify numerical
conditions for when a rule applies.

A rule must either have a scope or one or more conditions. Finally,
a rule with scope and conditions applies when all of them apply.
2018-06-13 11:20:38 +01:00
Ryan Ernst
a65b18f19d Core: Remove plain execute method on TransportAction (#30998)
TransportAction has many variants of execute. One of those variants
executes by returning a future, which is then often blocked on by
calling get(). This commit removes this variant of execute, instead
using a helper method for tests that want to block, or having tests
pass in a PlainActionFuture directly as a listener.

Co-authored-by: Simon Willnauer <simonw@apache.org>
2018-06-13 09:58:13 +02:00
Jason Tedor
0bfd18cc8b
Revert upgrade to Netty 4.1.25.Final (#31282)
This reverts upgrading to Netty 4.1.25.Final until we have a cleaner
solution to dealing with the object cleaner thread.
2018-06-12 19:26:18 -04:00
Tal Levy
2fae7a2ff5 fix explainlifecyclerequesttest 2018-06-12 14:47:39 -07:00
James Baiera
f88b9e83cf Fix Netty 4 Server Transport tests. Again. 2018-06-12 15:07:17 -04:00
Tal Levy
59ea407333 Merge branch 'master' into index-lifecycle 2018-06-12 10:54:58 -07:00
Costin Leau
bd81b95e95 [DOCS] Mark SQL feature as experimental 2018-06-12 19:46:08 +03:00
Lisa Cawley
ff0f4ece2f
[DOCS] Updates machine learning custom URL screenshots (#31222) 2018-06-12 08:57:11 -07:00
Jason Tedor
666997830e
Fix naming conventions check for XPackTestCase
This class needs to be abstract or it fails the naming convention check.
2018-06-12 11:34:40 -04:00
Jason Tedor
f738b7d9f8
Fix security Netty 4 transport tests
This test suite needs to filter out the object cleaner thread too so
this commit does that.
2018-06-12 11:31:43 -04:00
Albert Zaharovits
e988ace5f7
[DOCS] Clarify audit index settings when remote indexing (#30923) 2018-06-12 16:19:43 +03:00
Colin Goodheart-Smithe
ff80202353
Merge branch 'master' into index-lifecycle 2018-06-12 12:52:04 +01:00
Yuri Tceretian
ef453b31b4
Delete typos in SAML docs (#31199)
* Delete typos in SAML docs saml-guide.asciidoc
2018-06-12 14:45:27 +03:00
Dimitris Athanasiou
5f84e18c72
[ML][TEST] Mute tests using rules (#31204)
This is in preparation of pushing the new
rules design in the `ml-cpp` side. These
tests will be switched on again after merging
in the new rules implementation.
2018-06-12 11:36:26 +01:00
Ioannis Kakavas
b2e48c9fa7
Support RequestedAuthnContext (#31238)
* Support RequestedAuthnContext

This implements limited support for RequestedAuthnContext by :
- Allowing SP administrators to define a list of authnContextClassRef
to be included in the RequestedAuthnContext of a SAML Authn Request
- Veirifying that the authnContext in the incoming SAML Asertion's
AuthnStatement contains one of the requested authnContextClassRef
- Only EXACT comparison is supported as the semantics of validating
the incoming authnContextClassRef are deployment dependant and
require pre-established rules for MINIMUM, MAXIMUM and BETTER

Also adds necessary AuthnStatement validation as indicated by [1] and
[2]

[1] https://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf
    3.4.1.4, line 2250-2253
[2] https://kantarainitiative.github.io/SAMLprofiles/saml2int.html
    [SDP-IDP10]
2018-06-12 12:23:40 +03:00
Aditya Dhulipala
72f8199279 Validate xContentType in PutWatchRequest. (#31088)
Trying to post a new watch without any body currently results in a 
NullPointerException. This change fixes that by validating that 
Post and Put requests always have a body.

Closes #30057
2018-06-12 09:53:36 +02:00
Tal Levy
0541684df3 Merge branch 'master' into index-lifecycle 2018-06-11 17:20:05 -07:00
Nik Everett
0d9b78834f
LLClient: Support host selection (#30523)
Allows users of the Low Level REST client to specify which hosts a
request should be run on. They implement the  `NodeSelector` interface
or reuse a built in selector like `NOT_MASTER_ONLY` to chose which nodes
are valid. Using it looks like:
```
Request request = new Request("POST", "/foo/_search");
RequestOptions options = request.getOptions().toBuilder();
options.setNodeSelector(NodeSelector.NOT_MASTER_ONLY);
request.setOptions(options);
...
```

This introduces a new `Node` object which contains a `HttpHost` and the
metadata about the host. At this point that metadata is just `version`
and `roles` but I plan to add node attributes in a followup. The
canonical way to **get** this metadata is to use the `Sniffer` to pull
the information from the Elasticsearch cluster.

I've marked this as "breaking-java" because it breaks custom
implementations of `HostsSniffer` by renaming the interface to
`NodesSniffer` and by changing it from returning a `List<HttpHost>` to a
`List<Node>`. It *shouldn't* break anyone else though.

Because we expect to find it useful, this also implements `host_selector`
support to `do` statements in the yaml tests. Using it looks a little
like:

```
---
"example test":
  - skip:
      features: host_selector
  - do:
      host_selector:
        version: " - 7.0.0" # same syntax as skip
      apiname:
        something: true
```

The `do` section parses the `version` string into a host selector that
uses the same version comparison logic as the `skip` section. When the
`do` section is executed it passed the off to the `RestClient`, using
the `ElasticsearchHostsSniffer` to sniff the required metadata.

The idea is to use this in mixed version tests to target a specific
version of Elasticsearch so we can be sure about the deprecation
logging though we don't currently have any examples that need it. We do,
however, have at least one open pull request that requires something
like this to properly test it.

Closes #21888
2018-06-11 17:07:27 -04:00
Jason Tedor
563141c6c9
Upgrade to Netty 4.1.25.Final (#31232)
This commit upgrades us to Netty 4.1.25. This upgrade is more
challenging than past upgrades, all because of a new object cleaner
thread that they have added. This thread requires an additional security
permission (set context class loader, needed to avoid leaks in certain
scenarios). Additionally, there is not a clean way to shutdown this
thread which means that the thread can fail thread leak control during
tests. As such, we have to filter this thread from thread leak control.
2018-06-11 16:55:07 -04:00
Jason Tedor
cb952bd9ec
Enable custom credentials for core REST tests (#31235)
The core REST tests with security currently use a hardcoded username and
password. This is not amenable to running these tests in scenarios where
the user controls the creation of the cluster and owns the credentials
for this cluster. This commit enables running the core REST tests with
security with a custom username and password.
2018-06-11 16:53:40 -04:00
Tanguy Leroux
bf58660482
Remove all unused imports and fix CRLF (#31207)
The X-Pack opening and the recent other refactorings left a lot of 
unused imports in the codebase. This commit removes them all.
2018-06-11 15:12:12 +02:00
Tanguy Leroux
a1916658a9 [Tests] Fix self-referencing tests
This commit adapts some test after #31044 has been merged.
2018-06-11 12:45:27 +02:00
Colin Goodheart-Smithe
b865d0476d
Makes IndexLifecycleMetadata feature aware 2018-06-11 10:18:43 +01:00
Colin Goodheart-Smithe
9ee492a3f0
Merge branch 'master' into index-lifecycle 2018-06-11 09:47:26 +01:00
Jason Tedor
65c107b47d
Fix unknown licenses (#31223)
The goal of this commit is to address unknown licenses when producing
the dependencies info report. We have two different checks that we run
on licenses. The first check is whether or not we have stashed a copy of
the license text for a dependency in the repository. The second is to
map every dependency to a license type (e.g., BSD 3-clause). The problem
here is that the way we were handling licenses in the second check
differs from how we handle licenses in the first check. The first check
works by finding a license file with the name of the artifact followed
by the text -LICENSE.txt. Yet in some cases we allow mapping an artifact
name to another name used to check for the license (e.g., we map
lucene-.* to lucene, and opensaml-.* to shibboleth. The second check
understood the first way of looking for a license file but not the
second way. So in this commit we teach the second check about the
mappings from artifact names to license names. We do this by copying the
configuration from the dependencyLicenses task to the dependenciesInfo
task and then reusing the code from the first check in the second
check. There were some other challenges here though. For example,
dependenciesInfo was checking too many dependencies. For now, we should
only be checking direct dependencies and leaving transitive dependencies
from another org.elasticsearch artifact to that artifact (we want to do
this differently in a follow-up). We also want to disable
dependenciesInfo for projects that we do not publish, users only care
about licenses they might be exposed to if they use our assembled
products. With all of the changes in this commit we have eliminated all
unknown licenses. A follow-up will enforce that when we add a new
dependency it does not get mapped to unknown, these will be forbidden in
the future. Therefore, with this change and earlier changes are left
having no unknown licenses and two custom licenses; custom here means it
does not map to an SPDX license type. Those two licenses are xz and
ldapsdk. A future change will not allow additional custom licenses
unless they are explicitly whitelisted. This ensures that if a new
dependency is added it is mapped to an SPDX license or mapped to custom
because it does not have an SPDX license.
2018-06-09 07:28:41 -04:00
Julie Tibshirani
00b0e10063
Remove DocumentFieldMappers#simpleMatchToFullName. (#31041)
* Remove DocumentFieldMappers#simpleMatchToFullName, as it is duplicative of MapperService#simpleMatchToIndexNames.
* Rename MapperService#simpleMatchToIndexNames -> simpleMatchToFullName for consistency.
* Simplify EsIntegTestCase#assertConcreteMappingsOnAll to accept concrete fields instead of wildcard patterns.
2018-06-08 13:53:35 -07:00
Ioannis Kakavas
b26aae3915
Compliant SAML Response destination check (#31175)
Make SAML Response Destination check compliant

Only validate the Destination element of an incoming SAML Response
if Destination is present and the SAML Response is signed.
The standard [1] - 3.5.5.2 and [2] - 3.2.2 does mention that the
Destination element is optional and should only be verified when
the SAML Response is signed. Some Identity Provider implementations
are known to not set a Destination XML Attribute in their SAML
responses when those are not signed, so this change also aims to
enhance interoperability.

[1] https://docs.oasis-open.org/security/saml/v2.0/saml-bindings-2.0-os.pdf
[2] https://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf
2018-06-08 20:36:31 +03:00
Igor Motov
01140a3ad8
SQL: Make a single JDBC driver jar (#31012)
Replaces zip archive containing multiple jars with a single JDBC driver jar 
that shades all external dependencies.

Closes #29856
2018-06-08 10:15:28 -04:00