Commit Graph

3467 Commits

Author SHA1 Message Date
Chris Cho 1514d1a1be Change shard allocation filter property and api (#42602)
The current example is not working and a bit confused. This change tries
to match it with the sample of the watcher blog.
2019-06-04 12:30:39 -04:00
Ioannis Kakavas 440ec4d9f5
[Backport 7.x] OpenID Connect realm guide (#42836)
This commit adds a configuration guide for the newly introduced
OpenID Connect realm. The guide is similar to the style of the
SAML Guide and shares certain parts where applicable (role mapping)
It also contains a short section on how the realm can be used for
authenticating users without Kibana.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

Backport of #41423 and #42555
2019-06-04 14:08:41 +03:00
Tim Vernum 928f49992f
Don't require TLS for single node clusters (#42830)
This commit removes the TLS cluster join validator.

This validator existed to prevent v6.x nodes (which mandated
TLS) from joining an existing cluster of v5.x nodes (which did
not mandate TLS) unless the 6.x node (and by implication the
5.x nodes) was configured to use TLS.

Since 7.x nodes cannot talk to 5.x nodes, this validator is no longer
needed.

Removing the validator solves a problem where single node clusters
that were bound to local interfaces were incorrectly requiring TLS
when they recovered cluster state and joined their own cluster.

Backport of: #42826
2019-06-04 19:48:37 +10:00
Tim Vernum 8de3a88205
Log the status of security on license change (#42741)
Whether security is enabled/disabled is dependent on the combination
of the node settings and the cluster license.

This commit adds a license state listener that logs when the license
change causes security to switch state (or to be initialised).

This is primarily useful for diagnosing cluster formation issues.

Backport of: #42488
2019-06-04 14:25:43 +10:00
Tim Vernum 9035e61825
Detect when security index is closed (#42740)
If the security index is closed, it should be treated as unavailable
for security purposes.

Prior to 8.0 (or in a mixed cluster) a closed security index has
no routing data, which would cause a NPE in the cluster change
handler, and the index state would not be updated correctly.
This commit fixes that problem

Backport of: #42191
2019-06-04 14:25:20 +10:00
Benjamin Trent 87cc6a974c
[ML] [Data Frame] adding and modifying auditor messages (#42722) (#42818)
* [ML] [Data Frame] adding and modifying auditor messages

* Update DataFrameTransformTask.java
2019-06-03 19:49:58 -05:00
David Roberts b61202b0a8 [ML] Add a limit on line merging in find_file_structure (#42501)
When analysing a semi-structured text file the
find_file_structure endpoint merges lines to form
multi-line messages using the assumption that the
first line in each message contains the timestamp.
However, if the timestamp is misdetected then this
can lead to excessive numbers of lines being merged
to form massive messages.

This commit adds a line_merge_size_limit setting
(default 10000 characters) that halts the analysis
if a message bigger than this is created.  This
prevents significant CPU time being spent subsequently
trying to determine the internal structure of the
huge bogus messages.
2019-06-03 13:45:51 +01:00
Benjamin Trent 0253927ec4
[ML Data Frame] Refactor stop logic (#42644) (#42763)
* Revert "invalid test"

This reverts commit 9dd8b52c13c716918ff97e6527aaf43aefc4695d.

* Testing

* mend

* Revert "[ML Data Frame] Mute Data Frame tests"

This reverts commit 5d837fa312b0e41a77a65462667a2d92d1114567.

* Call onStop and onAbort outside atomic update

* Don’t update CS

* Tidying up

* Remove invalid test that asserted logic that has been removed

* Add stopped event

* Revert "Add stopped event"

This reverts commit 02ba992f4818bebd838e1c7678bd2e1cc090bfab.

* Adding check for STOPPED in saveState
2019-06-03 06:53:44 -05:00
David Roberts 10aca87389 [ML] Better detection of binary input in find_file_structure (#42707)
This change helps to prevent the situation where a binary
file uploaded to the find_file_structure endpoint is
detected as being text in the UTF-16 character set, and
then causes a large amount of CPU to be spent analysing
the bogus text structure.

The approach is to check the distribution of zero bytes
between odd and even file positions, on the grounds that
UTF-16BE or UTF16-LE would have a very skewed distribution.
2019-06-03 12:47:22 +01:00
Alan Woodward 2129d06643 Create client-only AnalyzeRequest/AnalyzeResponse classes (#42197)
This commit clones the existing AnalyzeRequest/AnalyzeResponse classes
to the high-level rest client, and adjusts request converters to use these new
classes.

This is a prerequisite to removing the Streamable interface from the internal
server version of these classes.
2019-06-03 09:46:36 +01:00
James Rodewig f51f8ed04c [DOCS] Remove unneeded options from `[source,sql]` code blocks (#42759)
In AsciiDoc, `subs="attributes,callouts,macros"` options were required
to render `include-tagged::` in a code block.

With elastic/docs#827, Elasticsearch Reference documentation migrated
from AsciiDoc to Asciidoctor.

In Asciidoctor, the `subs="attributes,callouts,macros"` options are no
longer needed to render `include-tagged::` in a code block. This commit
removes those unneeded options.

Resolves #41589
2019-05-31 13:05:13 -04:00
Benjamin Trent f22dcfb9da
[ML] [Data Frame] nesting group_by fields like other aggs (#42718) (#42760) 2019-05-31 10:55:35 -05:00
David Roberts 87ca762573 [ML] Add Kibana application privilege to data frame admin/user roles (#42757)
Data frame transforms are restricted by different roles to ML, but
share the ML UI.  To prevent the ML UI being hidden for users who
only have the data frame admin or user role, it is necessary to add
the ML Kibana application privilege to the backend data frame roles.
2019-05-31 15:41:59 +01:00
Jay Modi e687fd58fc
Re-enable token bwc tests (#42727)
This commit re-enables token bwc tests that run as part of the rolling
upgrade tests. These tests were muted while #42651 was being
backported.
2019-05-31 08:03:10 -06:00
Przemyslaw Gomulka d5061a151a
Remove suppresions for "unchecked" for hamcrest varargs methods Backport(41528) #42749
In hamcrest 2.1 warnings for unchecked varargs were fixed by hamcrest using @SafeVarargs for those matchers where this warning occurred.
This PR is aimed to remove these annotations when Matchers.contains ,Matchers.containsInAnyOrder or Matchers.hasItems was used
backport #41528
2019-05-31 13:58:49 +02:00
Przemysław Witek f6779de2b7
Increase maximum forecast interval to 10 years. (#41082) (#42710)
Increase the maximum duration to ~10 years (3650 days).
2019-05-31 06:19:47 +02:00
Jason Tedor 371cb9a8ce
Remove Log4j 1.2 API as a dependency (#42702)
We had this as a dependency for legacy dependencies that still needed
the Log4j 1.2 API. This appears to no longer be necessary, so this
commit removes this artifact as a dependency.

To remove this dependency, we had to fix a few places where we were
accidentally relying on Log4j 1.2 instead of Log4j 2 (easy to do, since
both APIs were on the compile-time classpath).

Finally, we can remove our custom Netty logger factory. This was needed
when we were on Log4j 1.2 and handled logging in our own unique
way. When we migrated to Log4j 2 we could have dropped this
dependency. However, even then Netty would still pick up Log4j 1.2 since
it was on the classpath, thus the advantage to removing this as a
dependency now.
2019-05-30 16:08:07 -04:00
Mark Vieira c1816354ed
[Backport] Improve build configuration time (#42674) 2019-05-30 10:29:42 -07:00
Benjamin Trent b5527b3278
[ML] [Data Frame] add support for weighted_avg agg (#42646) (#42714) 2019-05-30 12:05:35 -05:00
Jay Modi 711de2f59a
Make hashed token ids url safe (#42651)
This commit changes the way token ids are hashed so that the output is
url safe without requiring encoding. This follows the pattern that we
use for document ids that are autogenerated, see UUIDs and the
associated classes for additional details.
2019-05-30 10:44:41 -06:00
Ioannis Kakavas 7cabe8acc9 Fix refresh remote JWKS logic (#42662)
This change ensures that:

- We only attempt to refresh the remote JWKS when there is a
signature related error only ( BadJWSException instead of the
geric BadJOSEException )
- We do call OpenIDConnectAuthenticator#getUserClaims upon
successful refresh.
- We test this in OpenIdConnectAuthenticatorTests.

Without this fix, when using the OpenID Connect realm with a remote
JWKSet configured in `op.jwks_path`, the refresh would be triggered
for most configuration errors ( i.e. wrong value for `op.issuer` )
and the kibana wouldn't get a response and timeout since
`getUserClaims` wouldn't be called because
`ReloadableJWKSource#reloadAsync` wouldn't call `onResponse` on the
future.
2019-05-30 18:08:30 +03:00
Ioannis Kakavas 24a794fd6b Fix testTokenExpiry flaky test (#42585)
Test was using ClockMock#rewind passing the amount of nanoseconds
in order to "strip" nanos from the time value. This was intentional
as the expiration time of the UserToken doesn't have nanosecond
precision.
However, ClockMock#rewind doesn't support nanos either, so when it's
called with a TimeValue, it rewinds the clock by the TimeValue's
millis instead. This was causing the clock to go enough millis
before token expiration time and the test was passing. Once every
few hundred times though, the TimeValue by which we attempted to
rewind the clock only had nanos and no millis, so rewind moved the
clock back just a few millis, but still after expiration time.

This change moves the clock explicitly to the same instant as expiration,
using clock.setTime and disregarding nanos.
2019-05-30 07:53:56 +03:00
Igor Motov d2f9ccbe18 Geo: Refactor libs/geo parsers (#42549)
Refactors the WKT and GeoJSON parsers from an utility class into an
instantiatable objects. This is a preliminary step in
preparation for moving out coordinate validators from Geometry
constructors. This should allow us to make validators plugable.
2019-05-29 20:07:27 -04:00
David Kyle c5a410f68b [ML Data Frame] Set DF task state when stopping (#42516)
Set the state to stopped prior to persisting
2019-05-29 16:39:44 +01:00
Hendrik Muhs 345ff21ae5 [ML-DataFrame] rewrite start and stop to answer with acknowledged (#42589)
rewrite start and stop to answer with acknowledged

fixes #42450
2019-05-29 11:14:32 +02:00
Hendrik Muhs ace96a2b6e check position before and after latch (#42623)
check position before and after latch

#fixes 42084
2019-05-28 21:23:29 +02:00
David Kyle aea600fe7d [Ml Data Frame] Return bad_request on preview when config is invalid (#42447) 2019-05-28 15:36:50 +01:00
David Turner 746a2f41fd
Remove PRE_60_NODE_CHECKPOINT (#42531)
This commit removes the obsolete `PRE_60_NODE_CHECKPOINT` constant for dealing
with 5.x nodes' lack of sequence number support.

Backport of #42527
2019-05-28 12:25:53 +01:00
Daniel Mitterdorfer 635ce0ca6d
Mute AsyncTwoPhaseIndexerTests#testStateMachine() (#42610)
Relates #42084
Relates #42609
2019-05-28 10:20:22 +02:00
Nhat Nguyen 2077f9ffbc Reset mock transport service in CcrRetentionLeaseIT (#42600)
testRetentionLeaseIsAddedIfItDisappearsWhileFollowing does not reset the
mock transport service after test. Surviving transport interceptors from
that test can sneaky remove retention leases and make other tests fail.

Closes #39331
Closes #39509
Closes #41428
Closes #41679
Closes #41737
Closes #41756
2019-05-27 21:51:25 -04:00
Armin Braun a96606d962
Safer Wait for Snapshot Success in ClusterPrivilegeTests (#40943) (#42575)
* Safer Wait for Snapshot Success in ClusterPrivilegeTests

* The snapshot state returned by the API might become SUCCESS before it's fully removed from the cluster state.
  * We should fix this race in the transport API but it's not trivial and will be part of the incoming big round of refactoring the repository interaction, this added check fixes the test for now
* closes #38030
2019-05-27 12:08:20 +02:00
Armin Braun a5ca20a250
Some Cleanup in o.e.i.engine (#42278) (#42566)
* Some Cleanup in o.e.i.engine

* Remove dead code and parameters
* Reduce visibility in some obvious spots
* Add missing `assert`s (not that important here since the methods
themselves will probably be dead-code eliminated) but still
2019-05-27 11:04:54 +02:00
Nhat Nguyen 85e60850af Add debug log for retention leases (#42557)
We need more information to understand why CcrRetentionLeaseIT is
failing. This commit adds some debug log to retention leases and enables
them in CcrRetentionLeaseIT.
2019-05-26 16:04:47 -04:00
Nhat Nguyen d6e2f4a43e Enable recoveries trace log in CcrRetentionLeaseIT
Tracked #41679
2019-05-24 22:16:14 -04:00
Tanguy Leroux 6bec876682 Improve Close Index Response (#39687)
This changes the `CloseIndexResponse` so that it reports closing result
for each index. Shard failures or exception are also reported per index,
and the global acknowledgment flag is computed from the index results
only.

The response looks like:
```
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "indices" : {
    "docs" : {
      "closed" : true
    }
  }
}
```

The response reports shard failures like:
```
{
  "acknowledged" : false,
  "shards_acknowledged" : false,
  "indices" : {
    "docs-1" : {
      "closed" : true
    },
    "docs-2" : {
      "closed" : false,
      "shards" : {
        "1" : {
          "failures" : [
            {
              "shard" : 1,
              "index" : "docs-2",
              "status" : "BAD_REQUEST",
              "reason" : {
                "type" : "index_closed_exception",
                "reason" : "closed",
                "index_uuid" : "JFmQwr_aSPiZbkAH_KEF7A",
                "index" : "docs-2"
              }
            }
          ]
        }
      }
    },
    "docs-3" : {
      "closed" : true
    }
  }
}
```

Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
2019-05-24 21:57:55 -04:00
Julie Tibshirani 3a6c2525ca
Deprecate support for chained multi-fields. (#42330)
This PR contains a straight backport of #41926, and also updates the
migration documentation and deprecation info API for 7.x.
2019-05-24 15:55:06 -07:00
David Roberts 48dc0dca57 [ML] Use map and filter instead of flatMap in find_file_structure (#42534)
Using map and filter avoids the garbage from all the
Stream.of calls that flatMap necessitated. Performance
is better when there are masses of fields.
2019-05-24 20:12:06 +01:00
David Roberts 34de68b007 [ML] Fix possible race condition when closing an opening job (#42506)
This change fixes a race condition that would result in an
in-memory data structure becoming out-of-sync with persistent
tasks in cluster state.

If repeated often enough this could result in it being
impossible to open any ML jobs on the affected node, as the
master node would think the node had capacity to open another
job but the chosen node would error during the open sequence
due to its in-memory data structure being full.

The race could be triggered by opening a job and then closing
it a tiny fraction of a second later.  It is unlikely a user
of the UI could open and close the job that fast, but a script
or program calling the REST API could.

The nasty thing is, from the externally observable states and
stats everything would appear to be fine - the fast open then
close sequence would appear to leave the job in the closed
state.  It's only later that the leftovers in the in-memory
data structure might build up and cause a problem.
2019-05-24 20:11:58 +01:00
Hendrik Muhs 6d47ee9268 [ML-DataFrame] add support for fixed_interval, calendar_interval, remove interval (#42427)
* add support for fixed_interval, calendar_interval, remove interval

* adapt HLRC

* checkstyle

* add a hlrc to server test

* adapt yml test

* improve naming and doc

* improve interface and add test code for hlrc to server

* address review comments

* repair merge conflict

* fix date patterns

* address review comments

* remove assert for warning

* improve exception message

* use constants
2019-05-24 20:30:17 +02:00
Igor Motov e28a9e99c4 SQL: Moves the JTS-based tests suppression to Before (#42526)
Moves the test suppression from `ClassRule` to `Before`, where it is
properly handled in the CI build.

Fixes #42221
2019-05-24 13:58:53 -04:00
Hendrik Muhs 7cee294acf
[ML-DataFrame]backport dataframe changes from 42202, using client instead of transport (#42468)
backport dataframe changes from #42202, using client instead of transport
2019-05-24 11:05:30 +02:00
David Roberts f472186b9f [ML] Improve file structure finder timestamp format determination (#41948)
This change contains a major refactoring of the timestamp
format determination code used by the ML find file structure
endpoint.

Previously timestamp format determination was done separately
for each piece of text supplied to the timestamp format finder.
This had the drawback that it was not possible to distinguish
dd/MM and MM/dd in the case where both numbers were 12 or less.
In order to do this sensibly it is best to look across all the
available timestamps and see if one of the numbers is greater
than 12 in any of them.  This necessitates making the timestamp
format finder an instantiable class that can accumulate evidence
over time.

Another problem with the previous approach was that it was only
possible to override the timestamp format to one of a limited
set of timestamp formats.  There was no way out if a file to be
analysed had a timestamp that was sane yet not in the supported
set.  This is now changed to allow any timestamp format that can
be parsed by a combination of these Java date/time formats:
yy, yyyy, M, MM, MMM, MMMM, d, dd, EEE, EEEE, H, HH, h, mm, ss,
a, XX, XXX, zzz
Additionally S letter groups (fractional seconds) are supported
providing they occur after ss and separated from the ss by a dot,
comma or colon.  Spacing and punctuation is also permitted with
the exception of the question mark, newline and carriage return
characters, together with literal text enclosed in single quotes.

The full list of changes/improvements in this refactor is:

- Make TimestampFormatFinder an instantiable class
- Overrides must be specified in Java date/time format - Joda
  format is no longer accepted
- Joda timestamp formats in outputs are now derived from the
  determined or overridden Java timestamp formats, not stored
  separately
- Functionality for determining the "best" timestamp format in
  a set of lines has been moved from TextLogFileStructureFinder
  to TimestampFormatFinder, taking advantage of the fact that
  TimestampFormatFinder is now an instantiable class with state
- The functionality to quickly rule out some possible Grok
  patterns when looking for timestamp formats has been changed
  from using simple regular expressions to the much faster
  approach of using the Shift-And method of sub-string search,
  but using an "alphabet" consisting of just 1 (representing any
  digit) and 0 (representing non-digits)
- Timestamp format overrides are now much more flexible
- Timestamp format overrides that do not correspond to a built-in
  Grok pattern are mapped to a %{CUSTOM_TIMESTAMP} Grok pattern
  whose definition is included within the date processor in the
  ingest pipeline
- Grok patterns that correspond to multiple Java date/time
  patterns are now handled better - the Grok pattern is accepted
  as matching broadly, and the required set of Java date/time
  patterns is built up considering all observed samples
- As a result of the more flexible acceptance of Grok patterns,
  when looking for the "best" timestamp in a set of lines
  timestamps are considered different if they are preceded by
  a different sequence of punctuation characters (to prevent
  timestamps far into some lines being considered similar to
  timestamps near the beginning of other lines)
- Out-of-the-box Grok patterns that are considered now include
  %{DATE} and %{DATESTAMP}, which have indeterminate day/month
  ordering
- The order of day/month in formats with indeterminate day/month
  order is determined by considering all observed samples (plus
  the server locale if the observed samples still do not suggest
  an ordering)

Relates #38086
Closes #35137
Closes #35132
2019-05-24 09:10:08 +01:00
Tim Vernum 567c0d331f
Fix settings prefix for realm truststore password (#42413)
As part of #30241 realm settings were changed to be true affix
settings. In the process of this change, the "ssl." prefix was lost
from the realm truststore password. It should be:

    xpack.security.authc.realms.<type>.<name>.ssl.truststore.password

Due to a mismatch between the way we define SSL settings and  load SSL
contexts, there was no way to define this legacy password setting in a
realm config.

The settings validation would reject "ssl.truststore.password" but the
SSL service would ignore "truststore.password"

Backport of: #42336
2019-05-24 13:16:26 +10:00
Ryan Ernst a49bafc194
Split document and metadata fields in GetResult (#38373) (#42456)
This commit makes creators of GetField split the fields into document fields and metadata fields. It is part of larger refactoring that aims to remove the calls to static methods of MapperService related to metadata fields, as discussed in #24422.
2019-05-23 14:01:07 -07:00
Costin Leau a48125a9f7 Fix FROZEN indices backport 2019-05-23 21:30:41 +03:00
Costin Leau 9fdf4215dd Docs: Documentation for the upcoming SQL support of frozen indices (#41863)
(cherry picked from commit a3cc03eb1503df24c1706a721fcc9af38c3b2873)
(cherry picked from commit f42dcf2ffd7bd25f3f91aa6127515f393cd1860f)
2019-05-23 21:16:16 +03:00
Costin Leau d5f04d29c9 SQL: Add support for FROZEN indices (#41558)
Allow querying of FROZEN indices both through dedicated SQL grammar
extension:
> SELECT field FROM FROZEN index
and also through driver configuration parameter, namely:
> index.include.frozen: true/false

Fix #39390
Fix #39377

(cherry picked from commit 2445a933915f420c7f51e8505afa0a7978ce6b0f)
2019-05-23 21:16:16 +03:00
Zachary Tong 6d8a0e36ec Re-mute all ml_datafeed_crud rolling upgrade tests
AwaitsFix https://github.com/elastic/elasticsearch/issues/42258

Thought this was fixed, but throwing deprecation warnings at
an unexpected time so putting this back on mute until we
figure it out.
2019-05-23 09:50:27 -04:00
David Kyle a23257ce06 [ML Data Frame] Account for completed data frames in test (#42351)
When asserting on the checkpoint value if the DF has completed the checkpoint will be 1 else 0.
Similarly state may be started or indexing. Closes #42309
2019-05-23 14:05:09 +01:00
Jim Ferenczi b88e80ab89 Upgrade to Lucene 8.1.0 (#42214)
This commit upgrades to the GA release of Lucene 8.1.0
2019-05-23 11:46:45 +02:00
Jim Ferenczi 4ca5649a0d Upgrade to lucene 8.1.0-snapshot-e460356abe (#40952) 2019-05-23 11:45:33 +02:00
Mengwei Ding fa98cbe320
Add .code_internal-* index pattern to kibana user (#42247) (#42387) 2019-05-22 20:25:45 -07:00
Luca Cavanna c2af62455f Cut over SearchResponse and SearchTemplateResponse to Writeable (#41855)
Relates to #34389
2019-05-22 18:47:54 +02:00
Luca Cavanna 29c9bb9181 Clean up ShardId usage of Streamable (#41843)
ShardId already implements Writeable so there is no need for it to implement Streamable too. Also the readShardId static method can be
easily replaced with direct usages of the constructor that takes a
StreamInput as argument.
2019-05-22 18:47:54 +02:00
Yannick Welsch 5d8605c790 Fix testAutoFollowManyIndices
On a slow CI worker, the test was failing an assertion.

Closes #41234
2019-05-22 17:33:34 +02:00
David Kyle 075cc7c5cf [ML Data Frame] Persist data frame after state changes (#42347) 2019-05-22 15:40:40 +01:00
David Kyle f696769a39 Mute Data Frame integration tests
Relates to https://github.com/elastic/elasticsearch/issues/42344
2019-05-22 15:03:13 +01:00
Simon Willnauer a79cd77e5c Remove IndexShard dependency from Repository (#42213)
* Remove IndexShard dependency from Repository

In order to simplify repository testing especially for BlobStoreRepository
it's important to remove the dependency on IndexShard and reduce it to
Store and MapperService (in the snapshot case). This significantly reduces
the dependcy footprint for Repository and allows unittesting without starting
nodes or instantiate entire shard instances. This change deprecates the old
method signatures and adds a unittest for FileRepository to show the advantage
of this change.
In addition, the unittesting surfaced a bug where the internal file names that
are private to the repository were used in the recovery stats instead of the
target file names which makes it impossible to relate to the actual lucene files
in the recovery stats.

* don't delegate deprecated methods

* apply comments

* test
2019-05-22 14:27:11 +02:00
Alpar Torok eb1639c5fc TestClusters: Convert docs (#42100)
* TestClusters: Convert docs
2019-05-22 14:44:08 +03:00
Ioannis Kakavas aab97f1311 Fail early when rp.client_secret is missing in OIDC realm (#42256)
rp.client_secret is a required secure setting. Make sure we fail with
a SettingsException and a clear, actionable message when building
the realm, if the setting is missing.
2019-05-22 13:20:41 +03:00
Ioannis Kakavas ccdc0e6b3e Merge claims from userinfo and ID Token correctly (#42277)
Enhance the handling of merging the claims sets of the
ID Token and the UserInfo response. JsonObject#merge would throw a
runtime exception when attempting to merge two objects with the
same key and different values. This could happen for an OP that
returns different vales for the same claim in the ID Token and the
UserInfo response ( Google does that for profile claim ).
If a claim is contained in both sets, we attempt to merge the
values if they are objects or arrays, otherwise the ID Token claim
value takes presedence and overwrites the userinfo response.
2019-05-22 13:15:41 +03:00
Ioannis Kakavas 7af30345b4 Revert "mute failing filerealm hash caching tests (#42304)"
This reverts commit 39fbed1577.
2019-05-22 13:15:00 +03:00
Dimitris Athanasiou a6eb20ad35
[ML] Include node name when native controller cannot start process (#42225) (#42338)
This adds the node name where we fail to start a process via the native
controller to facilitate debugging as otherwise it might not be known
to which node the job was allocated.
2019-05-22 12:42:04 +03:00
Yannick Welsch 770d8e9e39 Remove usage of max_local_storage_nodes in test infrastructure (#41652)
Moves the test infrastructure away from using node.max_local_storage_nodes, allowing us in a
follow-up PR to deprecate this setting in 7.x and to remove it in 8.0.

This also changes the behavior of InternalTestCluster so that starting up nodes will not automatically
reuse data folders of previously stopped nodes. If this behavior is desired, it needs to be explicitly
done by passing the data path from the stopped node to the new node that is started.
2019-05-22 11:04:55 +02:00
Hendrik Muhs ad24231c1a [ML-DataFrame] validate group name to not contain invalid characters (#42292)
disallows of creating groupBy field with '[', ']', '>' in the name to be consistent with aggregations
2019-05-22 09:39:59 +02:00
Hendrik Muhs 3493f3b637 move latch await to doNextSearch (#42275)
move latch await to doNextSearch, fixes a race condition when the executor thread is faster than the coordinator thread

fixes #42084
2019-05-22 09:39:59 +02:00
Ioannis Kakavas 34dda75cdf Ensure SHA256 is not used in tests (#42289)
SHA256 was recently added to the Hasher class in order to be used
in the TokenService. A few tests were still using values() to get
the available algorithms from the Enum and it could happen that
SHA256 would be picked up by these.
This change adds an extra convenience method
(Hasher#getAvailableAlgoCacheHash) and enures that only this and
Hasher#getAvailableAlgoStoredHash are used for getting the list of
available password hashing algorithms in our tests.
2019-05-22 09:54:24 +03:00
Ioannis Kakavas cdf9485e33 Allow Kibana user to use the OpenID Connect APIs (#42305)
Add the manage_oidc privilege to the kibana user and to the role 
privileges list
2019-05-22 09:44:37 +03:00
Tim Vernum c5f191f6af
Add cluster restart for security on basic (#42217)
This performs a simple restart test to move a basic licensed
cluster from no security (the default) to security & transport TLS
enabled.

Backport of: #41933
2019-05-22 14:27:45 +10:00
Ed Savage 685a206891 Merge branch '7.x' of github.com:elastic/elasticsearch into 7.x 2019-05-21 19:14:17 +01:00
David Kyle 7e4d3c695b [ML Data Frame] Persist and restore checkpoint and position (#41942)
Persist and restore Data frame's current checkpoint and position
2019-05-21 18:57:13 +01:00
Ed Savage d97f4d5e28 [ML][TEST] Fix limits in AutodetectMemoryLimitIT (#42279)
Re-enable muted tests and accommodate recent backend changes
that result in higher memory usage being reported for a job
at the start of its life-cycle
2019-05-21 18:44:47 +01:00
Tal Levy 39fbed1577 mute failing filerealm hash caching tests (#42304)
some tests are failing after the introduction of #41792.

relates #42267 and #42289.
2019-05-21 10:40:14 -07:00
Dimitris Athanasiou a4e6fb4dd2
[ML] Fix logger declaration in ML plugins (#42222) (#42238)
This corrects what appears to have been a copy-paste error
where the logger for `MachineLearning` and `DataFrame` was wrongly
set to be that of `XPackPlugin`.
2019-05-21 18:03:24 +03:00
David Kyle 0fd42ce1f5
[ML Data Frame] Start directly data frame rather than via the scheduler (#42224)
Trigger indexer start directly to put the indexer in INDEXING state immediately
2019-05-21 15:48:45 +01:00
jimczi 0449869511 Fix unchecked warning in RollupIndexerIndexingTests#testSimpleDateHistoWithOverlappingDelay 2019-05-21 12:28:57 +02:00
David Kyle ffefc66260 Mute failing AsyncTwoPhaseIndexerTests
See https://github.com/elastic/elasticsearch/issues/42084
2019-05-21 10:24:46 +01:00
David Kyle 24144aead2
[ML] Complete the Data Frame task on stop (#41752) (#42063)
Wait for indexer to stop then complete the persistent task on stop.
If the wait_for_completion is true the request will not return until stopped.
2019-05-21 10:24:20 +01:00
Tim Vernum 7b3a9c7033
Do not refresh realm cache unless required (#42212)
If there are no realms that depend on the native role mapping store,
then changes should it should not perform any cache refresh.
A refresh with an empty realm array will refresh all realms.

This also fixes a spurious log warning that could occur if the
role mapping store was notified that the security index was recovered
before any realm were attached.

Backport of: #42169
2019-05-21 18:14:22 +10:00
Jim Ferenczi ec63160243 Fix max boundary for rollups job that use a delay (#42158)
Rollup jobs can define how long they should wait before rolling up new documents.
However if the delay is smaller or if it's not a multiple of the rollup interval
the job can create incomplete buckets because the max boundary for a job is computed
from the time when the job started rounded to the interval minus the delay. This change
fixes this computation by applying the delay substraction before the rounding in order to ensure
that we never create a boundary that falls in a middle of a bucket.
2019-05-21 08:48:53 +02:00
Zachary Tong 6ae6f57d39
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.

This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed.  And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit).  This arrangement is very
error-prone for users.

This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.

The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.

The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Alexander Reelsen c72c76b5ea Update to joda time 2.10.2 (#42199) 2019-05-20 16:58:54 +02:00
Ioannis Kakavas b4a413c4d0
Hash token values for storage (#41792) (#42220)
This commit changes how access tokens and refresh tokens are stored
in the tokens index.

Access token values are now hashed before being stored in the id
field of the `user_token` and before becoming part of the token
document id. Refresh token values are hashed before being stored
in the token field of the `refresh_token`. The tokens are hashed
without a salt value since these are v4 UUID values that have
enough entropy themselves. Both rainbow table attacks and offline
brute force attacks are impractical.

As a side effect of this change and in order to support multiple
concurrent refreshes as introduced in #39631, upon refreshing an
<access token, refresh token> pair, the superseding access token
and refresh tokens values are stored in the superseded token doc,
encrypted with a key that is derived from the superseded refresh
token. As such, subsequent requests to refresh the same token in
the predefined time window will return the same superseding access
token and refresh token values, without hitting the tokens index
(as this only stores hashes of the token values). AES in GCM
mode is used for encrypting the token values and the key
derivation from the superseded refresh token uses a small number
of iterations as it needs to be quick.

For backwards compatibility reasons, the new behavior is only
enabled when all nodes in a cluster are in the required version
so that old nodes can cope with the token values in a mixed
cluster during a rolling upgrade.
2019-05-20 17:55:29 +03:00
Jay Modi dbbdcea128
Update ciphers for TLSv1.3 and JDK11 if available (#42082)
This commit updates the default ciphers and TLS protocols that are used
when the runtime JDK supports them. New cipher support has been
introduced in JDK 11 and 12 along with performance fixes for AES GCM.
The ciphers are ordered with PFS ciphers being most preferred, then
AEAD ciphers, and finally those with mainstream hardware support. When
available stronger encryption is preferred for a given cipher.

This is a backport of #41385 and #41808. There are known JDK bugs with
TLSv1.3 that have been fixed in various versions. These are:

1. The JDK's bundled HttpsServer will endless loop under JDK11 and JDK
12.0 (Fixed in 12.0.1) based on the way the Apache HttpClient performs
a close (half close).
2. In all versions of JDK 11 and 12, the HttpsServer will endless loop
when certificates are not trusted or another handshake error occurs. An
email has been sent to the openjdk security-dev list and #38646 is open
to track this.
3. In JDK 11.0.2 and prior there is a race condition with session
resumption that leads to handshake errors when multiple concurrent
handshakes are going on between the same client and server. This bug
does not appear when client authentication is in use. This is
JDK-8213202, which was fixed in 11.0.3 and 12.0.
4. In JDK 11.0.2 and prior there is a bug where resumed TLS sessions do
not retain peer certificate information. This is JDK-8212885.

The way these issues are addressed is that the current java version is
checked and used to determine the supported protocols for tests that
provoke these issues.
2019-05-20 09:45:36 -04:00
Lisa Cawley fd2d4d761b [DOCS] Updates TLS configuration info (#41983) 2019-05-20 09:13:37 -04:00
Ed Savage 840af87a74 [ML] Temporarily muting failing tests
Muting a number of AutoDetectMemoryLimitIT tests to give CI a chance to
settle before easing in required backend changes.

relates elastic/ml-cpp#486
relates #42086
2019-05-19 08:29:50 -04:00
Ed Savage a68b04e47b [ML] Improve hard_limit audit message (#42086)
Improve the hard_limit memory audit message by reporting how many bytes
over the configured memory limit the job was at the point of the last
allocation failure.

Previously the model memory usage was reported, however this was
inaccurate and hence of limited use -  primarily because the total
memory used by the model can decrease significantly after the models
status is changed to hard_limit but before the model size stats are
reported from autodetect to ES.

While this PR contains the changes to the format of the hard_limit audit
message it is dependent on modifications to the ml-cpp backend to
send additional data fields in the model size stats message. These
changes will follow in a subsequent PR. It is worth noting that this PR
must be merged prior to the ml-cpp one, to keep CI tests happy.
2019-05-17 17:40:08 -04:00
Benjamin Trent f2447364fd
[ML] adds geo_centroid aggregation support to data frames (#42088) (#42094) 2019-05-17 16:51:05 -04:00
Igor Motov 076ca75ea5 SQL: Suppress geo tests failing on tr-TR locale (#42200)
Due to a bug in JTS WKT parser, JTS cannot parse most of WKT shapes if
the shape type is written in the lower case. For examples `point (1 2)`
is causing JTS inside H2GIS to fail on tr-TR locale as a result  of 
case-insensitive comparison.
2019-05-17 16:00:54 -04:00
David Roberts 226df35d96 [ML] Improve message misformation error in file structure finder (#42175)
This change replaces the extremely unfriendly message
"Number of messages analyzed must be positive" in the
case where the sample lines were incorrectly grouped
into just one message to an error that more helpfully
explains the likely root cause of the problem.
2019-05-16 18:29:38 +01:00
Ryan Ernst fa1d1d1f57 Deprecate the native realm migration tool (#42142)
The migrate tool was added when the native realm was created, to aid
users in converting from file realms that were per node, into the
cluster managed native realm. While this tool was useful at the time,
users should now be using the native realm directly. This commit
deprecates the tool, to be removed in a followup for 8.0.
2019-05-16 09:52:31 -04:00
Marios Trivyzas 7473742e6e
SQL: Fix issue regarding INTERVAL * number (#42014)
Interval * integer number is a valid operation which previously was
only supported for foldables (literals) and not when a field was
involved. That was because:

1. There was no common type returned for that combination
2. The `BinaryArithmeticOperation` was permitting the multiplication
(called by fold()) but the BinaryArithmeticProcessor didn't allow it

Moreover the error message for invalid arithmetic operations was wrong
because of the issue with the overloading methods of
`LoggerMessageFormat.format`.

Fixes: #41239
Fixes: #41200
(cherry picked from commit 91039bab12d3ef27d6eac9cdc891a3b3ad0c694d)
2019-05-15 16:06:55 -04:00
Tim Vernum 9191b02213
Enforce transport TLS on Basic with Security (#42150)
If a basic license enables security, then we should also enforce TLS
on the transport interface.

This was already the case for Standard/Gold/Platinum licenses.

For Basic, security defaults to disabled, so some of the process
around checking whether security is actuallY enabled is more complex
now that we need to account for basic licenses.
2019-05-15 13:59:27 -04:00
Igor Motov 70ea3cf847
SQL: Add initial geo support (#42031) (#42135)
Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. 

Queries that are supported as a result of this initial implementation

Metadata commands

- `DESCRIBE table`  - returns the correct column types `GEOMETRY` for geo shapes and geo points.
- `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions
- `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. 

Returning geoshapes and geopoints from elasticsearch

- `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console.
- `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation;

Using geopoints to elasticsearch

- The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. 
- `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query.

Limitations:

Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values.

Relates to #29872
Backport of #42031
2019-05-14 18:57:12 -05:00
Gordon Brown a85189a558
Remove toStepKeys from LifecycleAction (#41775)
The `toStepKeys()` method was only called in its own test case. The real
list of StepKeys that's used in action execution is generated from the
list of actual step objects returned by `toSteps()`.

This commit removes that method.
2019-05-10 16:06:42 -06:00
Benjamin Trent febee07dcc
[ML] adding pivot.max_search_page_size option for setting paging size (#41920) (#42079)
* [ML] adding pivot.size option for setting paging size

* Changing field name to address PR comments

* fixing ctor usage

* adjust hlrc for field name change
2019-05-10 13:22:31 -05:00
Benjamin Trent 0931815355
[ML] properly nesting objects in document source (#41901) (#42077)
* [ML] properly nesting objects in document source

* Throw exception on agg extraction failure, cause it to fail df

* throwing error to stop df if unsupported agg is found
2019-05-10 13:22:12 -05:00
Tal Levy 5640197632
Refactor TransportSingleShardAction to serialize Writeable responses (#41985) (#42040)
Previously, TransportSingleShardAction required constructing a new
empty response object. This response object's Streamable readFrom
was used. As part of the migration to Writeable, the interface here
was updated to leverage Writeable.Reader.

relates to #34389.
2019-05-09 22:08:31 -07:00
Benjamin Trent b23b06dded
[ML] verify that there are no duplicate leaf fields in aggs (#41895) (#42025)
* [ML] verify that there are no duplicate leaf fields in aggs

* addressing pr comments

* addressing PR comments

* optmizing duplication check
2019-05-09 14:29:10 -05:00
Benjamin Trent bf5a40c754
[ML] relax set upgrade mode test to match what is guaranteed (#41958) (#41979)
* [ML] relax set upgrade mode test to match what is guaranteed

* removing unused import
2019-05-09 14:28:50 -05:00
Lisa Cawley 9284a70ec8 [DOCS] Updates security configuration overview (#41982) 2019-05-09 07:50:51 -07:00
David Kyle c0d67919c8 Mute ApiKeyIntegTests
See https://github.com/elastic/elasticsearch/issues/41747
2019-05-09 13:24:52 +01:00
Zachary Tong f410f91f13 Cleanup RollupSearch exceptions, disallow partial results (#41272)
- msearch exceptions should be thrown directly instead of wrapping
in a RuntimeException
- Do not allow partial results (where some indices are missing), 
instead throw an exception if any index is missing
2019-05-08 12:38:42 -04:00
David Kyle ba9d2ccc1f [ML Data Frame] Set executing nodes in task actions (#41798)
Direct the task request to the node executing the task and also refactor the task responses
so all errors are returned and set the HTTP status code based on presence of errors.
2019-05-08 12:25:36 +01:00
Alpar Torok 711ace0533 Testclusters: support for security and convert example plugins (#41864)
testclusters detect from settings that security is enabled
if a user is not specified using the DSL introduced in this PR, a default one is created
the appropriate wait conditions are used authenticating with the first user defined in the DSL ( or the default user ).
an example DSL to create a user is user username:"test_user" password:"x-pack-test-password" role: "superuser" all keys are optional and default to the values shown in this example
2019-05-08 14:04:00 +03:00
Jason Tedor 8bea3c3a58
Enable trace logging in CCR retention lease tests
These tests are failing somewhat mysteriously, indicating that when we
renew retention leaess during a restore that our retention leases that
we added before starting the restore suddenly do not exist. To make
sense of this, this commit enables trace logging.
2019-05-07 22:44:55 -04:00
Marios Trivyzas d5b0badeb7
SQL: Remove CircuitBreaker from parser (#41835)
The CircuitBreaker was introduced as means of preventing a
`StackOverflowException` during the build of the AST by the parser.

The ANTLR4 grammar causes a weird behaviour for a Parser Listener.
The `enterEveryRule()` method is often called with a different parsing
context than the respective `exitEveryRule()`. This makes it difficult
to keep track of the tree's depth, and a custom Map was used as an
attempt of matching the contextes as they are encounter during `enter`
and during `exit` of the rules.

This approach had 2 important drawbacks:
1. It's hard to maintain this custom Map as the grammar changes.
2. The CircuitBreaker could often lead to false positives which caused
valid queries to return an Exception and prevent them from executing.

So, this removes completely the CircuitBreaker which is replaced be
a simple handling of the `StackOverflowException`

Fixes: #41471
(cherry picked from commit 1559a8e2dbd729138b52e89b7e80264c9f4ad1e7)
2019-05-07 23:25:37 +03:00
Ioannis Kakavas 58041f3fdb Remove op.name configuration setting (#41445)
This setting was not eventually used in the realm and thus can be
removed
2019-05-07 19:01:55 +03:00
Jason Tedor d7fd51a84e
Provide names for all artifact repositories (#41857)
This commit adds a name for each Maven and Ivy repository used in the
build.
2019-05-07 06:35:28 -04:00
Tim Vernum 3508b6c641
Log warning when unlicensed realms are skipped (#41828)
Because realms are configured at node startup, but license levels can
change dynamically, it is possible to have a running node that has a
particular realm type configured, but that realm is not permitted under
the current license.
In this case the realm is silently ignored during authentication.

This commit adds a warning in the elasticsearch logs if authentication
fails, and there are realms that have been skipped due to licensing.
This message is not intended to imply that the realms could (or would)
have successfully authenticated the user, but they may help reduce
confusion about why authentication failed if the caller was expecting
the authentication to be handled by a particular realm that is in fact
unlicensed.

Backport of: #41778
2019-05-07 09:55:48 +10:00
Ryan Ernst 6fd8924c5a Switch run task to use real distro (#41590)
The run task is supposed to run elasticsearch with the given plugin or
module. However, for modules, this is most realistic if using the full
distribution. This commit changes the run setup to use the default or
oss as appropriate.
2019-05-06 12:34:07 -07:00
Benjamin Trent 50fc27e9a0
[ML] addresses preview bug, and adds check to PUT (#41803) (#41850) 2019-05-06 10:56:26 -05:00
Tim Brooks 927013426a
Read multiple TLS packets in one read call (#41820)
This is related to #27260. Currently we have a single read buffer that
is no larger than a single TLS packet. This prevents us from reading
multiple TLS packets in a single socket read call. This commit modifies
our TLS work to support reading similar to the plaintext case. The data
will be copied to a (potentially) recycled TLS packet-sized buffer for
interaction with the SSLEngine.
2019-05-06 09:51:32 -06:00
Daniel Mitterdorfer 8580053818
Mute PermissionsIT#testWhen[...]ByILMPolicy (#41859)
Relates #41440
Relates #41858
2019-05-06 16:15:37 +02:00
Hendrik Muhs d54a921032 remove unused import 2019-05-06 10:14:35 +02:00
Hendrik Muhs 0c03707704 [ML-DataFrame] reset/clear the position after indexer is done (#41736)
reset/clear the position after indexer is done
2019-05-06 09:41:51 +02:00
Tim Vernum ee84038699
Update security acknowledgement messages for basic (#41825)
When applying a license update, we provide "acknowledgement messages"
that indicate which features will be affected by the change in license.

This commit updates the messages that are provided when installing a
basic license, so that they reflect the changes made to the security
features that are included in that license type.

Backport of: #41776
2019-05-06 16:40:38 +10:00
Hicham Mallah 4a88da70c5 Add index name to cluster block exception (#41489)
Updates the error message to reveal the index name that is causing it.

Closes #40870
2019-05-04 19:11:59 -04:00
Jason Tedor f4da98ca3d
Use a proper repository for ml-cpp artifacts (#41817)
This switches the strategy used to download machine learning artifacts
from a manual download through S3 to using an Ivy repository on top of
S3. This gives us all the benefits of Gradle dependency resolution
including local caching.
2019-05-04 12:44:19 -04:00
Benjamin Trent b69e28177b
[ML] rewriting stats gathering to use callbacks instead of a latch (#41793) (#41804) 2019-05-03 18:18:27 -05:00
Nhat Nguyen c7924014fa
Verify consistency of version and source in disruption tests (#41614) (#41661)
With this change, we will verify the consistency of version and source
(besides id, seq_no, and term) of live documents between shard copies
at the end of disruption tests.
2019-05-03 18:47:14 -04:00
Jason Tedor 241c4ef97a
Use https for artifact locations
This commit switches to using https for some artifact locations.
2019-05-03 16:15:48 -04:00
Hendrik Muhs 00af42fefe move checkpoints into x-pack core and introduce base classes for data frame tests (#41783)
move checkpoints into x-pack core and introduce base classes for data frame tests
2019-05-03 14:16:25 +02:00
Hendrik Muhs befe2a45b9 [ML-DataFrame] refactor pivot to only take the pivot config (#41763)
refactor pivot class to only take the config at construction, other parameters are passed in as part of
method that require them
2019-05-03 13:37:51 +02:00
Jason Tedor d0f071236a
Simplify filtering addresses on interfaces (#41758)
This commit is a refactoring of how we filter addresses on
interfaces. In particular, we refactor all of these methods into a
common private method. We also change the order of logic to first check
if an address matches our filter and then check if the interface is
up. This is to possibly avoid problems we are seeing where devices are
flapping up and down while we are checking for loopback addresses. We do
not expect the loopback device to flap up and down so by reversing the
logic here we avoid that problem on CI machines. Finally, we expand the
error message when this does occur so that we know which device is
flapping.
2019-05-02 16:36:27 -04:00
Benjamin Trent 33b4032fab
[ML] Correct indexer state on task re-allocation (#41724) (#41751) 2019-05-02 12:01:59 -05:00
Benjamin Trent a70f796edd
[ML] fix array oob in IDGenerator and adjust format for mapping (#41703) (#41717)
* [ML] fix array oob in IDGenerator and adjust format for mapping

* Update DataFramePivotRestIT.java
2019-05-02 11:09:42 -05:00
Tim Brooks b4bcbf9f64
Support http read timeouts for transport-nio (#41466)
This is related to #27260. Currently there is a setting
http.read_timeout that allows users to define a read timeout for the
http transport. This commit implements support for this functionality
with the transport-nio plugin. The behavior here is that a repeating
task will be scheduled for the interval defined. If there have been
no requests received since the last run and there are no inflight
requests, the channel will be closed.
2019-05-02 09:48:52 -06:00
Benjamin Trent a92c06ae09
[ML] Refactor NativeStorageProvider to enable reuse (#41414) (#41746)
* [ML] Refactor NativeStorageProvider to enable reuse

Moves `NativeStorageProvider` as a machine learning component
so that it can be reused for other job types. Also, we now
pass the persistent task description as unique identifier which
avoids conflicts between jobs of different type but with same ids.

* Adding nativeStorageProvider as component

Since `TransportForecastJobAction` is expected to get injected a `NativeStorageProvider` class, we need to make sure that it is a constructed component, as it does not have a zero parametered, public ctor.
2019-05-02 09:46:22 -05:00
Hendrik Muhs be7ec5a47a simplify indexer by moving members to base class (#41741)
simplify indexer by moving members to base class
2019-05-02 16:08:08 +02:00
Christoph Büscher bf1a70f242 Mute RejectedExecutionTests#testHistoryAndTriggeredOnRejection 2019-05-02 11:50:04 +02:00
Jason Tedor 0870523489
Fix compilation in SecurityMocks
This commit fixes compilation in SecurityMocks from what appears to be
some merge conflicts that were not resolved adequately.
2019-05-01 14:29:33 -04:00
Jason Tedor f500d727cf
Resolve conflicts in AuthenticationServiceTests
This commit resolves some merge conflicts that arose in
AuthenticationServiceTests after a rebase.
2019-05-01 14:20:58 -04:00
Jason Tedor 942a1445f3
Fix reference to 7.1 in security token tests
This version should be referencing 7.2 rather than 7.1, due to some
changes in timing of the token service changes.
2019-05-01 14:00:35 -04:00
Ioannis Kakavas 8426130553
Add negative tests for security features in basic
Assert that API Keys, Tokens, DLS/FLS do not work in basic
2019-05-01 14:00:32 -04:00
Tim Vernum 3589ca8493
Add test for security on basic license.
This is modelled on the qa test for TLS on basic.

It starts a cluster on basic with security & performs a number of
security related checks.
It also performs those same checks on a trial license.
2019-05-01 14:00:29 -04:00
Tim Vernum 0ee16d0115
Security on Basic License
This adds support for using security on a basic license.
It includes:

- AllowedRealmType.NATIVE realms (reserved, native, file)
- Roles / RBAC
- TLS (already supported)

It does not support:

- Audit
- IP filters
- Token Service & API Keys
- Advanced realms (AD, LDAP, SAML, etc)
- Advanced roles (DLS, FLS)
- Pluggable security

As with trial licences, security is disabled by default.

This commit does not include any new automated tests, but existing tests have been updated.
2019-05-01 14:00:25 -04:00
Jason Tedor 7f3ab4524f
Bump 7.x branch to version 7.2.0
This commit adds the 7.2.0 version constant to the 7.x branch, and bumps
BWC logic accordingly.
2019-05-01 13:38:57 -04:00
Albert Zaharovits 990be1f806
Security Tokens moved to a new separate index (#40742)
This commit introduces the `.security-tokens` and `.security-tokens-7`
alias-index pair. Because index snapshotting is at the index level granularity
(ie you cannot snapshot a subset of an index) snapshoting .`security` had
the undesirable effect of storing ephemeral security tokens. The changes
herein address this issue by moving tokens "seamlessly" (without user
intervention) to another index, so that a "Security Backup" (ie snapshot of
`.security`) would not be bloated by ephemeral data.
2019-05-01 14:53:56 +03:00
Jason Tedor 0b46a62f6b
Drop distinction in entries for keystore (#41701)
Today we allow adding entries from a file or from a string, yet we
internally maintain this distinction such that if you try to add a value
from a file for a setting that expects a string or add a value from a
string for a setting that expects a file, you will have a bad time. This
causes a pain for operators such that for each setting they need to know
this difference. Yet, we do not need to maintain this distinction
internally as they are bytes after all. This commit removes that
distinction and includes logic to upgrade legacy keystores.
2019-05-01 07:02:04 -04:00
Jake Landis 7ac3283509
Fix Watcher deadlock that can cause in-abilty to index documents. (#41418) (#41684)
This commit removes the usage of the `BulkProcessor` to write history documents
and delete triggered watches on a `EsRejectedExecutionException`. Since the
exception could be handled on the write thread, the write thread can be blocked
waiting on watcher threads (due to a synchronous method). This is problematic
since those watcher threads can be blocked waiting on write threads.

This commit also moves the handling of the exception to the generic threadpool
to avoid submitting write requests from the write thread pool.

fixes #41390
2019-04-30 16:13:24 -05:00
Tom Veasey b3f4533e1c [ML] Update for model selection change and disable temporarily (#41482) (#41682) 2019-04-30 15:47:54 -05:00
James Rodewig 66d8549083 [DOCS] Move block delimiter and anchor in 'Configuring Security' for Asciidoctor migration (#41604) 2019-04-30 16:41:26 -04:00
Nhat Nguyen 887f3f2c83 Simplify initialization of max_seq_no of updates (#41161)
Today we choose to initialize max_seq_no_of_updates on primaries only so
we can deal with a situation where a primary is on an old node (before
6.5) which does not have MUS while replicas on new nodes (6.5+).
However, this strategy is quite complex and can lead to bugs (for
example #40249) since we have to assign a correct value (not too low) to
MSU in all possible situations (before recovering from translog,
restoring history on promotion, and handing off relocation).

Fortunately, we don't have to deal with this BWC in 7.0+ since all nodes
in the cluster should have MSU. This change simplifies the
initialization of MSU by always assigning it a correct value in the
constructor of Engine regardless of whether it's a replica or primary.

Relates #33842
2019-04-30 15:14:52 -04:00
David Kyle f737b05ad1 Mute CcrRetentionLeaseIT.testForgetFollower
https://github.com/elastic/elasticsearch/issues/39850
2019-04-30 09:55:16 +01:00
Tim Brooks df3ef66294
Remove dedicated SSL network write buffer (#41654)
This is related to #27260. Currently for the SSLDriver we allocate a
dedicated network write buffer and encrypt the data into that buffer one
buffer at a time. This requires constantly switching between encrypting
and flushing. This commit adds a dedicated outbound buffer for SSL
operations that will internally allocate new packet sized buffers as
they are need (for writing encrypted data). This allows us to totally
encrypt an operation before writing it to the network. Eventually it can
be hooked up to buffer recycling.

This commit also backports the following commit:

Handle WRAP ops during SSL read

It is possible that a WRAP operation can occur while decrypting
handshake data in TLS 1.3. The SSLDriver does not currently handle this
well as it does not have access to the outbound buffer during read call.
This commit moves the buffer into the Driver to fix this issue. Data
wrapped during a read call will be queued for writing after the read
call is complete.
2019-04-29 17:59:13 -06:00
Benjamin Trent 92a820bc1a
[ML] Add bucket_script agg support to data frames (#41594) (#41639) 2019-04-29 10:14:17 -05:00
David Kyle 1a6ffb2644 Mute ClusterPrivilegeTests.testThatSnapshotAndRestore
Tracked in #38030
2019-04-29 16:45:01 +10:00
Yogesh Gaikwad c0d40ae4ca
Remove deprecated stashWithOrigin calls and use the alternative (#40847) (#41562)
This commit removes the deprecated `stashWithOrigin` and
modifies its usage to use the alternative.
2019-04-28 21:25:42 +10:00
Benjamin Trent a0990ca239
[ML] cleanup + adding description field to transforms (#41554) (#41605)
* [ML] cleanup + adding description field to transforms

* making description length have a max of 1k
2019-04-26 16:50:59 -05:00
Chris Earle 858e7f4a62
[7.x] [Monitoring] Add `usage` mapping for `monitoring-kibana` index (#40899) (#41601)
Backports the usage change to 7.x. A separate backport is needed to
change the version for 6.7, which will complete this backport.
2019-04-26 16:44:03 -04:00
James Rodewig d548901855 [DOCS] Add space to fix Asciidoctor output (#41579) 2019-04-26 12:13:33 -04:00
Alpar Torok 335f2bf102 Testclsuters: convert plugins qa projects (#41496)
Add testclusters support for files in keystore and convert qa subprojects within plugins.
2019-04-26 08:57:52 -07:00
Benjamin Trent 3ccb48e516
[ML] data frame, verify primary shards are active for configs index before task start (#41551) (#41580) 2019-04-26 10:23:43 -05:00
Armin Braun aad33121d8
Async Snapshot Repository Deletes (#40144) (#41571)
Motivated by slow snapshot deletes reported in e.g. #39656 and the fact that these likely are a contributing factor to repositories accumulating stale files over time when deletes fail to finish in time and are interrupted before they can complete.

* Makes snapshot deletion async and parallelizes some steps of the delete process that can be safely run concurrently via the snapshot thread poll
   * I did not take the biggest potential speedup step here and parallelize the shard file deletion because that's probably better handled by moving to bulk deletes where possible (and can still be parallelized via the snapshot pool where it isn't). Also, I wanted to keep the size of the PR manageable.
* See https://github.com/elastic/elasticsearch/pull/39656#issuecomment-470492106
* Also, as a side effect this gives the `SnapshotResiliencyTests` a little more coverage for master failover scenarios (since parallel access to a blob store repository during deletes is now possible since a delete isn't a single task anymore).
* By adding a `ThreadPool` reference to the repository this also lays the groundwork to parallelizing shard snapshot uploads to improve the situation reported in #39657
2019-04-26 15:36:09 +02:00
Benjamin Trent 4836ff7bcd
[ML] add multi node integ tests for data frames (#41508) (#41552)
* [ML] adding native-multi-node-integTests for data frames'

* addressing streaming issues

* formatting fixes

* Addressing PR comments
2019-04-26 07:18:49 -05:00
Tim Brooks 1f8ff052a1
Revert "Remove dedicated SSL network write buffer (#41283)"
This reverts commit f65a86c258.
2019-04-25 18:39:25 -06:00
Nhat Nguyen c4cb0507b4 Mute testDriverConfigurationWithSSLInURL
Tracked at #41557
2019-04-25 17:52:57 -04:00
Costin Leau b288b88ba0 SQL: Use field caps inside DESCRIBE TABLE as well (#41377)
Thanks to #34071, there is enough information in field caps to infer
the table structure and thus use the same API consistently across the
IndexResolver.

(cherry picked from commit f99946943a3350206b6bca774b2f060f41a787b3)
2019-04-25 23:41:17 +03:00
Tim Brooks f65a86c258
Remove dedicated SSL network write buffer (#41283)
This is related to #27260. Currently for the SSLDriver we allocate a
dedicated network write buffer and encrypt the data into that buffer one
buffer at a time. This requires constantly switching between encrypting
and flushing. This commit adds a dedicated outbound buffer for SSL
operations that will internally allocate new packet sized buffers as
they are need (for writing encrypted data). This allows us to totally
encrypt an operation before writing it to the network. Eventually it can
be hooked up to buffer recycling.
2019-04-25 14:30:54 -06:00
Christoph Büscher 52495843cc [Docs] Fix common word repetitions (#39703) 2019-04-25 20:47:47 +02:00
Tim Brooks 6d7110edf5
SSLDriver can transition to CLOSED in handshake (#41458)
TLS 1.3 changes to the SSLEngine introduced a scenario where a UNWRAP
call during a handshake can consume a close notify alerty without
throwing an exception. This means that we continue down a codepath where
we assert that we are still in handshaking mode. Transitioning to closed
from handshaking is a valid scenario. This commit removes this
assertion.
2019-04-25 12:02:17 -06:00
RomainGeffraye a02f5544fe Update configuring-ldap-realm.asciidoc (#40427) 2019-04-25 13:29:46 -04:00
Benjamin Trent 08843ba62b
[ML] Adds progress reporting for transforms (#41278) (#41529)
* [ML] Adds progress reporting for transforms

* fixing after master merge

* Addressing PR comments

* removing unused imports

* Adjusting afterKey handling and percentage to be 100*

* Making sure it is a linked hashmap for serialization

* removing unused import

* addressing PR comments

* removing unused import

* simplifying code, only storing total docs and decrementing

* adjusting for rewrite

* removing initial progress gathering from executor
2019-04-25 11:23:12 -05:00
Jim Ferenczi 6184efaff6
Handle unmapped fields in _field_caps API (#34071) (#41426)
Today the `_field_caps` API returns the list of indices where a field
is present only if this field has different types within the requested indices.
However if the request is an index pattern (or an alias, or both...) there
is no way to infer the indices if the response contains only fields that have
the same type in all indices. This commit changes the response to always return
the list of indices in the response. It also adds a way to retrieve unmapped field
in a specific section per field called `unmapped`. This section is created for each field
that is present in some indices but not all if the parameter `include_unmapped` is set to
true in the request (defaults to false).
2019-04-25 18:13:48 +02:00
Armin Braun 40aef2b8aa
Introduce Delegating ActionListener Wrappers (#40129) (#41527)
* Introduce Delegating ActionListener Wrappers
* Dry up use cases of ActionListener that simply pass through the response or exception to another listener
2019-04-25 16:05:04 +02:00
Albert Zaharovits fe5789ada1 Fix Has Privilege API check on restricted indices (#41226)
The Has Privileges API allows to tap into the authorization process, to validate
privileges without actually running the operations to be authorized. This commit
fixes a bug, in which the Has Privilege API returned spurious results when checking
for index privileges over restricted indices (currently .security, .security-6,
.security-7). The actual authorization process is not affected by the bug.
2019-04-25 12:03:27 +03:00
Ryan Ernst 7e3875d781 Upgrade hamcrest to 2.1 (#41464)
hamcrest has some improvements in newer versions, like FileMatchers
that make assertions regarding file exists cleaner. This commit upgrades
to the latest version of hamcrest so we can start using new and improved
matchers.
2019-04-24 23:40:03 -07:00
Zachary Tong ec5dd0594f Disallow null/empty or duplicate composite sources (#41359)
Adds some validation to prevent duplicate source names from being
used in the composite agg.

Also refactored to use a ConstructingObjectParser and removed the
private ctor and setter for sources, making it mandatory.
2019-04-24 13:23:31 -04:00
Hendrik Muhs 13d720f4a7 [ML-DataFrame] Replace Streamable w/ Writeable (#41477)
replace usages of Streamable with Writeable

Relates to #36176
2019-04-24 17:17:34 +02:00
Benjamin Trent 07d36fdb23
[ML] refactoring the ML plugin to use the common auditor code (#41419) (#41485) 2019-04-24 09:56:59 -05:00
James Rodewig 122e727d28 [DOCS] Fix broken link to elasticsearch-php security page 2019-04-23 12:43:40 -04:00
Benjamin Trent e2f8ffdde8
[ML][Data Frame] Moving destination creation to _start (#41416) (#41433)
* [ML][Data Frame] Moving destination creation to _start

* slight refactor of DataFrameAuditor constructor
2019-04-23 09:32:57 -05:00
Martijn Laarman 85b9dc18a7 fix #35262 define deprecations of API's as a whole and urls (#39063)
* fix #35262 define deprecations of API's as a whole and urls

* document hot threads deprecated paths

* deprecate scroll_id as part of the URL, documented only as part of the body which is a safer behaviour as well

* use version numbers up to patch version

* rest spec parser picks up deprecated paths as paths too

(cherry picked from commit 7e06023e7603b7584bfd9ee4e8a1ccd82c208ce7)
2019-04-23 14:28:36 +02:00
Marios Trivyzas e991175776
SQL: Implement IIF(<cond>, <result1>, <result2>) (#41420)
Implement a more trivial case of the CASE expression which is
expressed as a traditional function with 2 or 3 arguments. e.g.:

IIF(a = 1, 'one', 'many')
IIF(a > 0, 'positive')
Closes: #40917

(cherry picked from commit add02f4f553ad472026dcc1eaa84245a0558a4b0)
2019-04-23 16:31:25 +03:00
Marios Trivyzas 67d4e399c2
SQL: Implement CASE... WHEN... THEN... ELSE... END (#41349)
Implement the ANSI SQL CASE expression which provides the if/else
functionality common to most programming languages.

The CASE expression can have multiple WHEN branches and becomes a
powerful tool for SQL queries as it can be used in SELECT, WHERE,
GROUP BY, HAVING and ORDER BY clauses.

Closes: #36200
(cherry picked from commit 8b2577406f47ae60d15803058921d128390af0b6)
2019-04-22 19:26:56 +03:00
Dimitris Athanasiou eb2295ac81
[7.1][ML] Refactor autodetect service into its own class (#41378) (#41409)
This also improves aims to improve the corresponding unit tests
with regard to readability and maintainability.
2019-04-22 17:42:13 +03:00
Ioannis Kakavas b8d054e73b OpenID Connect realm settings and rest API docs (#40740)
This commit adds the relevant docs for the OpenID Connect
realm settings and the REST APIs that are exposed.
2019-04-22 15:50:18 +03:00
Albert Zaharovits c3e0ae24d3
Fix role mapping DN field wildcards for users with NULL DNs (#41343)
The `DistinguishedNamePredicate`, used for matching users to role mapping
expressions, should handle users with null DNs. But it fails to do so (and this is
a NPE bug), if the role mapping expression contains a lucene regexp or a wildcard.

The fix simplifies `DistinguishedNamePredicate` to not handle null DNs at all, and
instead use the `ExpressionModel#NULL_PREDICATE` for the DN field, just like
any other missing user field.
2019-04-22 10:25:24 +03:00
Jason Tedor 5c40fc9ba5
Remove script engine from X-Pack plugin (#41387)
The X-Pack plugin implements ScriptEngine yet it does not actually
implement any of the methods on the interface, effectively making this a
no-op. This commit removes this interface from the X-Pack plugin.
2019-04-20 08:23:20 -04:00
Jason Tedor 21bf2fe3c4
Reduce security permissions in CCR plugin (#41391)
It looks like these permissions were copy/pasted from another plugin yet
almost none of these permissions are needed for the CCR plugin. This
commit removes all these unneeded permissions from the CCR plugin.
2019-04-20 08:21:59 -04:00
Marios Trivyzas 7a34ba35f7
SQL: Fix bug with optimization of null related conditionals (#41355)
The SimplifyConditional rule is removing NULL literals from those
functions to simplify their evaluation. This happens in the Optimizer
and a new instance of the conditional function is generated. Previously,
the dataType was not set properly (defaulted to DataType.NULL) for
those new instances and since the resolveType() wasn't called again
it resulted in returning always null.

E.g.:

SELECT COALESCE(null, 'foo', null, 'bar')

COALESCE(null, 'foo', null, 'bar')
-----------------
null

This issue was not visible before because the tests always used an alias
for the conditional function which caused the resolveType() to be
called which sets the dataType properly.

E.g.:

SELECT COALESCE(null, 'foo', null, 'bar') as c

c
-----------------
foo

(cherry picked from commit c39980a65dd593363f1d8d1b038b26cb0ce02aaf)
2019-04-19 19:04:32 +03:00
Lisa Cawley c03394c236 [DOCS] Fixes deprecation notice in pagerduty action (#41362) 2019-04-18 17:33:07 -07:00
Costin Leau 5f497d4274 SQL: Predicate diff takes into account all values (#41346)
Fix bug in predicate subtraction that caused the evaluation to be
skipped on the first mismatch instead of evaluating the whole list. In
some cases this caused not only an incorrect result but one that kept on
growing causing the engine to bail

Fix #40835

(cherry picked from commit bd2b33d6eaca616a5acd846204e2d12f905854d4)
2019-04-18 22:44:50 +03:00
Andrei Stefan cfed5d65be SQL: fix *SecurityIT tests by covering edge case scenarios when audit file rolls over at midnight (#41328)
* Handle the scenario where assertLogs() is not called from a test method
but the audit rolling file rolls over.
* Use a local boolean variable instead of the static one to account for
assertBusy() code block possibly being called multiple times and having
different execution paths.

(cherry picked from commit 6f642196cbab90079c610097befc794746170df1)
2019-04-18 21:24:18 +03:00
Mark Vieira 0227ac5690
Fix issue with subproject test task dependencies (#41321) (#41351) 2019-04-18 11:15:34 -07:00
Adrien Grand 86e56590a7 Revert "Disable CcrRetentionLeaseIT#testRetentionLeasesAreNotBeingRenewedAfterRecoveryCompletes."
This reverts commit 343039e200.
2019-04-18 11:31:00 +02:00
Adrien Grand 343039e200 Disable CcrRetentionLeaseIT#testRetentionLeasesAreNotBeingRenewedAfterRecoveryCompletes.
Relates #39331.
2019-04-18 11:29:11 +02:00
Gordon Brown 66366d0307
Extract template management from Watcher (#41169)
This commit extracts the template management from Watcher into an
abstract class, so that templates and lifecycle policies can be managed
in the same way across multiple plugins. This will be useful for SLM, as
well as potentially ILM and any other plugins which need to manage index
templates.
2019-04-17 13:42:36 -06:00
Zachary Tong 7e62ff2823 [Rollup] Validate timezones based on rules not string comparision (#36237)
The date_histogram internally converts obsolete timezones (such as
"Canada/Mountain") into their modern equivalent ("America/Edmonton").
But rollup just stored the TZ as provided by the user.

When checking the TZ for query validation we used a string comparison,
which would fail due to the date_histo's upgrading behavior.

Instead, we should convert both to a TimeZone object and check if their
rules are compatible.
2019-04-17 13:46:44 -04:00
Iana Bondarska e090176f17 [ML] Exclude analysis fields with core field names from anomaly results (#41093)
Added "_index", "_type", "_id" to list of reserved fields.

Closes #39406
2019-04-17 16:08:03 +01:00
David Kyle 1d2365f5b6 [ML-DataFrame] Refactorings and tidying (#41248)
Remove unnecessary generic params from SingleGroupSource
and unused code from the HLRC
2019-04-17 14:58:26 +01:00
David Kyle 711d2545aa [ML-DataFrame] Resolve random test failure using deterministic name (#41262) 2019-04-17 09:04:20 +01:00
Marios Trivyzas 3fd081528d
SQL: Allow current_date/time/timestamp to be also used as a function escape pattern (#41254)
CURRENT_DATE/CURRENT_TIME/CURRENT_TIMESTAMP can be used as SQL keywords
(without parentheses) and therefore there is a special rule in the
grammar to accommodate this.

Previously, this rule was also catching the parenthesised version of those functions too,
not allowing the {fn <functionName>()} to be used. E.g.:
{fn current_time(2)} or {fn current_timestamp()}

Now, the grammar rule catches only the keyword versions and all the parenthesised
versions go through the normal function resolution. As a consequence the validation
of the precision is moved from the parser lever (ExpressionBuilder) to the function
implementations.

Fixes: #41240
(cherry picked from commit bfbc9f140144b5a35aa29008b58bf58074419853)
2019-04-17 10:13:16 +03:00
Yogesh Gaikwad 0d1178fca6
put mapping authorization for alias with write-index and multiple read indices (#40834) (#41287)
When the same alias points to multiple indices we can write to only one index
with `is_write_index` value `true`. The special handling in case of the put
mapping request(to resolve authorized indices) has a check on indices size
for a concrete index. If multiple indices existed then it marked the request
as unauthorized.

The check has been modified to consider write index flag and only when the
requested index matches with the one with write index alias, the alias is considered
for authorization.

Closes #40831
2019-04-17 14:25:33 +10:00
Yogesh Gaikwad 6a552c05fe
Use alias name from rollover request to query indices stats (#40774) (#41284)
In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes #40771
2019-04-17 14:15:05 +10:00
Costin Leau 85912b89fe SQL: Fix LIMIT bug in agg sorting (#41258)
When specifying a limit over an agg sorting, the limit will be pushed
down to the grouping which affects the custom sorting. This commit fixes
that and restricts the limit only to sorting.

Fix #40984

(cherry picked from commit da3726528d9011b05c0677ece6d11558994eccd9)
2019-04-16 22:40:41 +03:00
Marios Trivyzas c4ffd75805 SQL: Translate MIN/MAX on keyword fields as FIRST/LAST (#41247)
Although the translation rule was implemented in the `Optimizer`,
the rule was not added in the list of rules to be executed.

Relates to #41195
Follows #37936


(cherry picked from commit f426a339b77af6008d41cc000c9199fe384e9269)
2019-04-16 20:57:23 +03:00
Hendrik Muhs 02247cc7df [ML-DataFrame] adapt page size on circuit breaker responses (#41149)
handle circuit breaker response and adapt page size to reduce memory pressure, reduce preview buckets to 100, initial page size to 500
2019-04-16 19:49:43 +02:00
Costin Leau 9bf8bd40ae SQL: Tweak pattern matching in SYS TABLES (#41243)
Yet another improvement to SYS TABLES on differentiating between table
types specified as '%' and '' while maintaining legacy support for null

Fix #40775

(cherry picked from commit 6dbca5edd335eb1da8e7825389a15e5fe45397d4)
2019-04-16 19:48:45 +03:00
Gordon Brown ec8709e831
Check allocation rules are cleared after ILM Shrink (#41170)
Adds some checks to make sure that the allocation rules that ILM adds
before a shrink are cleared after the shrink is complete
2019-04-16 09:25:51 -06:00
David Kyle 116167df55 [ML] Write header to autodetect before it is visible to other calls (#41085) 2019-04-16 13:51:29 +01:00
Shaunak Kashyap 750db02b54
Expand beats_system role privileges (#40876) (#41232)
Traditionally we have [recommended](https://www.elastic.co/guide/en/beats/filebeat/current/monitoring.html) that Beats send their monitoring data to the **production** Elasticsearch cluster. Beats do this by calling the `POST _monitoring/bulk` API. When Security is enabled this API call requires the `cluster:admin/xpack/monitoring/bulk` privilege. The built-in `beats_system` role has this privilege.

[Going forward](https://github.com/elastic/beats/pull/9260), Beats will be able to send their monitoring data directly to the **monitoring** Elasticsearch cluster. Beats will do this by calling the regular `POST _bulk` API. When Security is enabled this API call requires the `indices:data/write/bulk` privilege. Further, the call has to be able to create any indices that don't exist.

This PR expands the built-in `beats_system` role's privileges. Specifically, it adds index-level `write` and `create_index` privileges for `.monitoring-beats-*` indices. 

This will allow Beats users to continue using the `beats_system` role for the new direct monitoring route when Security is enabled.
2019-04-15 20:17:05 -07:00
Tim Brooks 56c00eecbc
Remove string usages of old transport settings (#41207)
This is related to #36652. We intend to deprecate a number of transport
settings in 7.x and remove them in 8.0. This commit removes the string
usages of these settings.
2019-04-15 16:54:24 -06:00
Gordon Brown 7e59794ced
Log every use of ILM Move to Step API (#41171)
Usage of the ILM Move to Step API can result in some very odd
situations, and for diagnosing problems arising from these situations it
would be nice to have a record of when this API was called with what
parameters.

Also, adds a dedicated logger for TransportMoveToStepAction, 
rather than using the (deprecated) inherited one.
2019-04-15 16:20:37 -06:00
Martijn van Groningen f56b2ecb37
Remove xpack dependencies from qa rest modules (#41134) (7.x backport) (#41202)
This commit removes xpack dependencies of many xpack qa modules.
(for some qa modules this will require some more work)

The reason behind this change is that qa rest modules should not depend
on the x-pack plugins, because the plugins are an implementation detail and
the tests should only know about the rest interface and qa cluster that is
being tested.

Also some qa modules rely on xpack plugins and hlrc (which is a valid
dependency for rest qa tests) creates a cyclic dependency and this is
something that we should avoid.  Also Eclipse can't handle gradle cyclic
dependencies (see #41064).

* don't copy xpack-core's plugin property into the test resource of qa
modules. Otherwise installing security manager fails, because it tries
to find the XPackPlugin class.
2019-04-15 19:14:43 +02:00
David Kyle 2b539f8347 [ML DataFrame] Data Frame stop all (#41156)
Wild card support for the data frame stop API
2019-04-15 15:04:28 +01:00
David Roberts 3f00c29adb [ML] Allow xpack.ml.max_machine_memory_percent higher than 100% (#41193)
Values higher than 100% are now allowed to accommodate use
cases where swapping has been determined to be acceptable.
Anomaly detector jobs only use their full model memory
during background persistence, and this is deliberately
staggered, so with large numbers of jobs few will generally
be persisting state at the same time.  Settings higher than
available memory are only recommended for OEM type
situations where a wrapper tightly controls the types of
jobs that can be created, and each job alone is considerably
smaller than what each node can handle.
2019-04-15 14:37:46 +01:00
Alpar Torok 9e94db1b6a Mute failing test
Tracked in #41172
2019-04-15 15:27:32 +03:00
Ioannis Kakavas fe9442b05b
Add an OpenID Connect authentication realm (#40674) (#41178)
This commit adds an OpenID Connect authentication realm to
elasticsearch. Elasticsearch (with the assistance of kibana or
another web component) acts as an OpenID Connect Relying
Party and supports the Authorization Code Grant and Implicit
flows as described in http://ela.st/oidc-spec. It adds support
for consuming and verifying signed ID Tokens, both RP
initiated and 3rd party initiated Single Sign on and RP
initiated signle logout.
It also adds an OpenID Connect Provider in the idp-fixture to
be used for the associated integration tests.

This is a backport of #40674
2019-04-15 12:41:16 +03:00
Christoph Büscher 2980a6c70f Clarify some ToXContent implementations behaviour (#41000)
This change adds either ToXContentObject or ToXContentFragment to classes
directly implementing ToXContent currently. This helps in reasoning about
whether those implementations output full xcontent object or just fragments.

Relates to #16347
2019-04-15 09:42:08 +02:00
Gordon Brown c8bc4ab003
Improve Watcher test framework resiliency (#40658)
It is possible for the watches tracked by ScheduleTriggerEngineMock to
get out of sync with the Watches in the ScheduleTriggerEngine
production code, which can lead to watches failing to run.

This commit:

1. Changes TimeWarp to try to run the watch on all schedulers, rather than stopping after one which claims to have the watch registered. This reduces the impact of desynchronization between the mocking code and the backing production code.
2. Makes ScheduleTriggerEngineMock respect pauses of execution again. This is necessary to prevent duplicate watch invocations due to the above change.
3. Tweaks how watches are registered in ScheduleTriggerEngineMock to prevent race conditions due to concurrent modification.
4. Tweaks WatcherConcreteIndexTests to use TimeWarp instead of waiting for watches to be triggered, as TimeWarp is more reliable and accomplishes the same goal.
2019-04-12 16:53:50 -06:00
Yogesh Gaikwad 47ba45732d
Find and use non local IPv4 address while testing IP filtering (#40234) (#41141)
For pattern "n:localhost" PatternRule#isLocalhost() matches
any local address, loopback address.
[Note: I think for "localhost" this should not consider IP address
as a match when they are bound to network interfaces. It should just
be loopback address check unless the intent is to match all local addresses.
This class is adopted from Netty3 and I am not sure if this is intended
behavior or maybe I am missing something]

For now I have fixed this assuming the PatternRule#isLocalhost check is
correct by avoiding use of local address to check address denied.

Closes #40194
2019-04-13 04:37:25 +10:00
Lee Hinman 7b190609ab
(7.x) Use environment settings instead of state settings for Watcher config (#41158)
Backport of (#41087)

* Use environment settings instead of state settings for Watcher config

Prior to this we used the settings from cluster state to see whether ILM was
enabled of disabled, however, these settings don't accurately reflect the
`xpack.ilm.enabled` setting in `elasticsearch.yml`.

This commit changes to using the `Environment` settings, which correctly reflect
the ILM enabled setting.

Resolves #41042
2019-04-12 12:15:44 -06:00
Gordon Brown ef310886a7
Add deprecation check for ILM poll interval <1s (#41096)
ILM poll intervals of less than 1 second will not be allowed, so add a
deprecation check for that.

Even though I'm pretty sure zero production clusters will do this, it's
best to be thorough.
2019-04-12 10:43:04 -06:00
Nik Everett c379206c1e
Fix some documentation urls in rest-api-spec (#40618) (#41145)
Fixes some documentation urls in the rest-api-spec. Some of these URLs
pointed to 404s and a few others pointed to deprecated documentation
when we have better documentation now. I'm not consistent about `master`
vs `current` because we're not consistent in other places and I think we
should solve all of those at once with something a little more
automatic.
2019-04-12 10:11:14 -04:00
Hendrik Muhs 3df6798c4c Rollup/DataFrame: disallow partial results (#41114)
disallow partial results in rollup and data frame, after this change the client throws an error directly
replacing the previous runtime exception thrown, allowing better error handling in implementations.
2019-04-12 07:31:22 +02:00
Benjamin Trent 05cf53934a
[ML] checking if p-tasks metadata is null before updating state (#41091) (#41123)
* [ML] checking if p-tasks metadata is null before updating state

* Adding test that validates fix

* removing debug println
2019-04-11 13:54:41 -05:00
Benjamin Trent 9e32e36799
[ML] fixing test related to #40963 (#41074) (#41116) 2019-04-11 11:19:56 -05:00
Armin Braun 233df6b73b
Make Transport Shard Bulk Action Async (#39793) (#41112)
This is a dependency of #39504

Motivation:
By refactoring `TransportShardBulkAction#shardOperationOnPrimary` to async, we enable using `DeterministicTaskQueue` based tests to run indexing operations. This was previously impossible since we were blocking on the `write` thread until the `update` thread finished the mapping update.
With this change, the mapping update will trigger a new task in the `write` queue instead.
This change significantly enhances the amount of coverage we get from `SnapshotResiliencyTests` (and other potential future tests) when it comes to tracking down concurrency issues with distributed state machines.

The logical change is effectively all in `TransportShardBulkAction`, the rest of the changes is then simply mechanically moving the caller code and tests to being async and passing the `ActionListener` down.

Since the move to async would've added more parameters to the `private static` steps in this logic, I decided to inline and dry up (between delete and update) the logic as much as I could instead of passing the listener + wait-consumer down through all of them.
2019-04-11 16:01:52 +02:00
Dimitris Athanasiou 736c7285d4
[ML] Fix scroll size comparison in DatafeedUpdate.isNoop (#41056) (#41079)
Note this does not affect users as the method is only used in tests.
2019-04-11 11:29:51 +03:00
Jason Tedor 24446ceae0
Add packaging to cluster stats response (#41048)
This commit adds a packaging_types field to the cluster stats response
that outlines the build flavors and types present in a cluster.
2019-04-10 13:47:19 -04:00
Hendrik Muhs c37b127a07 fix a timing issue: isFinished is used for a busy loop in testing, (#41055)
test: ensure state is persisted before the isFinished is changed

fixes #41046
2019-04-10 18:47:34 +02:00
Martijn van Groningen 1eff8976a8
Deprecate AbstractHlrc* and AbstractHlrcStreamable* base test classes (#41014)
* moved hlrc parsing tests from xpack to hlrc module and removed dependency on hlrc from xpack core

* deprecated old base test class

* added deprecated jdoc tag

* split test between xpack-core part and hlrc part

* added lang-mustache test dependency, this previously came in via
hlrc dependency.

* added hlrc dependency on a qa module

* duplicated ClusterPrivilegeName class in xpack-core, since x-pack
core no longer has a dependency on hlrc.

* replace ClusterPrivilegeName usages with string literals

* moved tests to dedicated to hlrc packages in order to remove Hlrc part from the name and make sure to use imports instead of full qualified class where possible

* remove ESTestCase. from method invocation and use method directly,
because these tests indirectly extend from ESTestCase
2019-04-10 16:29:17 +02:00
Chris Earle 64a05e522f
Properly handle Monitoring exporters all disabled (#40920) (#41043)
When monitoring exporters are all disabled, which must be done
explicitly, _and_ monitoring collection is enabled, then
any call to `_xpack/monitoring/_bulk` will create a task that
never closes _and_ ES collection will stop happening because
a semaphore is never marked as completed.

This also simplifies the async `ExportBulk` code by removing the
third step (second async step, `close`) entirely because it was
entirely unnecessary by both implementations.
2019-04-10 10:09:28 -04:00
Christoph Büscher d00d3f4afa Mute DataFrameTransformCheckpointTests#testGetBehind 2019-04-10 15:55:09 +02:00
Przemysław Witek f5014ace64
[ML] Add validation that rejects duplicate detectors in PutJobAction (#40967) (#41072)
* [ML] Add validation that rejects duplicate detectors in PutJobAction

Closes #39704

* Add YML integration test for duplicate detectors fix.

* Use "== false" comparison rather than "!" operator.

* Refine error message to sound more natural.

* Put job description in square brackets in the error message.

* Use the new validation in ValidateJobConfigAction.

* Exclude YML tests for new validation from permission tests.
2019-04-10 15:43:35 +02:00
Albert Zaharovits adf3393a4e
Deprecate permission over aliases (#38059) (#41060)
This PR generates deprecation log entries for each Role Descriptor,
used for building a Role, when the Role Descriptor grants more privileges
for an alias compared to an index that the alias points to. This is done in
preparation for the removal of the ability to define privileges over aliases.
There is one log entry for each "role descriptor name"-"alias name" pair.
On such a notice, the administrator is expected to modify the Role Descriptor
definition so that the name pattern for index names does not cover aliases.

Caveats:
* Role Descriptors that are not used in any authorization process,
either because they are not mapped to any user or the user they are mapped to
is not used by clients, are not be checked.
* Role Descriptors are merged when building the effective Role that is used in
the authorization process. Therefore some Role Descriptors can overlap others,
so even if one matches aliases in a deprecated way, and it is reported as such,
it is not at risk from the breaking behavior in the current role mapping configuration
and index-alias configuration. It is still reported because it is a best practice to
change its definition, or remove offending aliases.
2019-04-10 15:02:33 +03:00
Costin Leau 38f471ae1c SQL: Change schema calls to empty set (#41034)
As empty string has a certain meaning, the JDBC driver returns an empty
set instead for better client compatibility.

Fix #41028

(cherry picked from commit 4cbafa585b7a514eb6c156606dd516324cd3980a)
2019-04-10 13:14:12 +03:00
Hendrik Muhs f9018ab11b [ML-DataFrame] create checkpoints on every new run (#40725)
Use the checkpoint service to create a checkpoint on every new run. Expose checkpoints stats on _stats endpoint.
2019-04-10 09:14:11 +02:00
Andrei Stefan 0157ebf947 Increase the interval filtering for CURRENT_DATE/TODAY tests (#40999)
(cherry picked from commit a5911330aecee90da5401e468b9976f66a2556f0)
2019-04-10 08:35:09 +03:00
Jason Tedor bb6f060f74
Add log message to forget follower test
This commit adds a log message to help debug failures in a forget
follower test.
2019-04-09 23:33:29 -04:00
Julie Tibshirani 0d5f86a001 Mute RollupIndexerStateTests#testIndexing.
Tracked in #41046.
2019-04-09 17:17:04 -07:00
Julie Tibshirani 0702c72151 Mute DataFrameGetAndGetStatsIT#testGetPersistedStatsWithoutTask.
Tracked in #40963.
2019-04-09 16:39:16 -07:00
Julie Tibshirani 21c5d7e95f Mute CcrRetentionLeaseIT#testRetentionLeasesAreNotBeingRenewedAfterRecoveryCompletes.
Tracked in #39331.
2019-04-09 16:08:44 -07:00
Julie Tibshirani cbae617898 Mute IndexFollowingIT#testFollowIndex as we await a fix.
Tracked in #41037.
2019-04-09 14:56:37 -07:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Jason Tedor ebba9393c1
Fix unsafe publication of invalid license enforcer (#40985)
The invalid license enforced is exposed to the cluster state update
thread (via the license state listener) before the constructor has
finished. This violates the JLS for safe publication of an object, and
means there is a concurrency bug lurking here. This commit addresses
this by avoiding publication of the invalid license enforcer before the
constructor has returned.
2019-04-09 13:51:37 -04:00
Armin Braun e71db0531e
Fix Race in AsyncTwoPhaseIndexerTests.testStateMachine (#40947) (#41013)
* The step is incremented by the listner in `org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexerTests.MockIndexer#onFinish` after isFinished is set to true, but the test only waited for `isFinished`,
fixed by calling `isFinished` last
* Also made `step` volatile since we are reading it from different thread from the one incrementing it
* Closes #40946
2019-04-09 19:40:01 +02:00
Costin Leau 2ac514b909 SQL: Fix catalog filtering in SYS COLUMNS (#40583)
Properly treat '%' as a wildcard for catalog filtering instead of doing
a straight string match.
Table filtering now considers aliases as well.
Add escaping char for LIKE queries with user defined params
Fix monotony of ORDINAL_POSITION
Add integration test for SYS COLUMNS - currently running only inside
single_node since the cluster name is test dependent.
Add pattern unescaping for index names

Fix #40582

(cherry picked from commit 8e61b77d3f849661b7175544f471119042fe9551)
2019-04-09 18:44:27 +03:00
Mark Vieira 64c98c632a
Ignore failing tests (#40994) 2019-04-08 21:17:49 -07:00
Jason Tedor 26d8ecfe07
Fix unsafe publication in opt-out query cache (#40957)
This opt-out query cache has an unsafe publication issue, where the
cache is exposed to another thread (namely the cluster state update
thread) before the constructor has finished execution. This exposes the
opt-out query cache to concurrency bugs. This commit addresses this by
ensuring that the opt-out query cache is not registered as a listener
for license state changes until after the constructor has returned.
2019-04-08 16:11:20 -04:00
Marios Trivyzas 2206491277 SQL: Refactor args verification of In & conditionals (#40916)
Move verification of arguments for Conditional functions and IN
from `Verifier` to the `resolveType()` method of the functions.

(cherry picked from commit 241644aac57baee1eb128b993ee410c7d08172a5)
2019-04-08 17:27:19 +02:00
Mark Vieira 2569fb60de Avoid sharing source directories as it breaks intellij (#40877)
* Avoid sharing source directories as it breaks intellij
* Subprojects share main project output classes directory
* Fix jar hell
* Fix sql security with ssl integ tests
* Relax dependency ordering rule so we don't explode on cycles
2019-04-08 17:26:46 +03:00
Costin Leau af874635ba SQL: Prefer resultSets over exceptions in metadata (#40641)
Changed the JDBC metadata to return empty results sets instead of
throwing SQLFeatureNotSupported as it seems a more safer/compatible
approach for consumers.

Fix #40533

(cherry picked from commit ef2d2527c2b5140556fd477e7ff6ea36966684da)
2019-04-08 16:45:34 +03:00
David Kyle fb5d7cf237 Mute AsyncTwoPhaseIndexerTests.testStateMachine 2019-04-08 11:14:35 +01:00
Marios Trivyzas ddf17dfb1e
SQL: Small code improvements of Pipes & Processors (#40909)
- Remove superfluous methods that are already
defined in superclasses.
- Improve tests for null folding on conditionals

(cherry picked from commit 67f9404f5004362e569353d1e950ffe5d7a9ab6e)
2019-04-08 10:14:13 +02:00
Tim Vernum 26c63e0115
Add test for HTTP and Transport TLS on basic license (#40932)
This adds a new security/qa test for TLS on a basic license.

It starts a 2 node cluster with a basic license, and TLS enabled
on both HTTP and Transport, and verifies the license type, x-pack
SSL usage and SSL certificates API.

It also upgrades the cluster to a trial license and performs that
same set of checks (to ensure that clusters with basic license
and TLS enabled can be upgraded to a higher feature license)

Backport of: #40714
2019-04-08 13:23:12 +10:00
Hendrik Muhs d5fcbf2f4a refactor onStart and onFinish to take runnables and executed them guarded by state (#40855)
refactor onStart and onFinish to take action listeners and execute them when indexer is in indexing state.
2019-04-07 21:46:37 +02:00
Jay Modi f34663282c
Update apache httpclient to version 4.5.8 (#40875)
This change updates our version of httpclient to version 4.5.8, which
contains the fix for HTTPCLIENT-1968, which is a bug where the client
started re-writing paths that contained encoded reserved characters
with their unreserved form.
2019-04-05 13:48:10 -06:00
David Turner 2ff19bc1b7
Use Writeable for TransportReplAction derivatives (#40905)
Relates #34389, backport of #40894.
2019-04-05 19:10:10 +01:00
Benjamin Trent a8dbb07546
[ML] Changes default destination index field mapping and adds scripted_metric agg (#40750) (#40846)
* [ML] Allowing destination index mappings to have dynamic types, adds script_metric agg

* Making dynamic|source mapping explicit
2019-04-05 11:34:20 -05:00
Brandon Kobel eb656244bf Add Kibana application privileges for monitoring and ml reserved roles (#40651)
* Add Kibana application privileges for monitoring and ml reserved roles

* Adding test for kibana-.kibana application explicitly

* Whoa there, fat fingered kibana and application...

* And I copied something from monitoring I shouldn't have...

* And actually doing what Yogesh recommended...
2019-04-05 07:44:29 -07:00
Benjamin Trent 665f0d81aa
[ML] refactoring start task a bit, removing unused code (#40798) (#40845) 2019-04-05 09:01:01 -05:00
Tim Vernum 920024e06d
Remove dynamic objects from security index (#40715)
The security index had a few "object" types with

   "dynamic": true

However, this automatically creates a mapping for each field that is
created within those objects. This means that types are dynamically
inferred and "locked in" for future updates.

Instead we want "dynamic": false which will allow us to store a range
of fields in these nested objects and retrieve them from the source,
without creating mapping types for those fields.

Backport of: #40499
2019-04-05 18:28:09 +11:00
Martijn van Groningen 809a5f13a4
Make -try xlint warning disabled by default. (#40833)
Many gradle projects specifically use the -try exclude flag, because
there are many cases where auto-closeable resource ignore is never
referenced in body of corresponding try statement. Suppressing this
warning specifically in each case that it happens using
`@SuppressWarnings("try")` would be very verbose.

This change removes `-try` from any gradle project and adds it to the
build plugin. Also this change removes exclude flags from gradle projects
that is already specified in build plugin (for example -deprecation).

Relates to #40366
2019-04-05 08:02:26 +02:00
Jim Ferenczi a15f55b2de Rollup ignores time_zone on date histogram (#40844)
When translating the original aggregation for the rollup indices,
the timezone of the date histogram is validated against the rollup
job but the value is not copied in the newly created date_histogram.
2019-04-04 21:16:50 +02:00
Gordon Brown 5347dec55e
Allow ILM to stop if indices have nonexistent policies (#40820)
Prior to this PR, there is a bug in ILM which does not allow ILM to stop
if one or more indices have an index.lifecycle.name which refers to
a policy that does not exist - the operation_mode will be stuck as
STOPPING until either the policy is created or the nonexistent
policy is removed from those indices.

This change allows ILM to stop in this case and makes the logging more
clear as to why ILM is not stopping.
2019-04-04 11:46:21 -06:00
Tim Vernum 9d34164ce6
Add build utility to check cluster health over ssl (#40713)
By default, in integ tests we wait for the standalone cluster to start
by using the ant Get task to retrieve the cluster health endpoint.
However the ant task has no facilities for customising the trusted
CAs for a https resource, so if the integ test cluster has TLS enabled
on the http interface (using a custom CA) we need a separate utility
for that purpose.

Backport of: #40573
2019-04-04 21:44:03 +11:00
David Roberts d16f86f7ab [ML] Add created_by info to usage stats (#40518)
This change adds information about which UI path
(if any) created ML anomaly detector jobs to the
stats returned by the _xpack/usage endpoint.

Counts for the following possibilities are expected:

* ml_module_apache_access
* ml_module_apm_transaction
* ml_module_auditbeat_process_docker
* ml_module_auditbeat_process_hosts
* ml_module_nginx_access
* ml_module_sample
* multi_metric_wizard
* population_wizard
* single_metric_wizard
* unknown

The "unknown" count is for jobs that do not have a
created_by setting in their custom_settings.

Closes #38403
2019-04-04 10:55:20 +01:00
Marios Trivyzas f3c207d27b SQL: Implement CURRENT_TIME/CURTIME functions (#40662)
After `TIME` SQL data type is introduced, implement
`CURRENT_TIME/CURTIME` functions similarly to CURRENT_TIMESTAMP
that return the system's current time (only, without the date part).

Closes: #40468
(cherry picked from commit 9feede781409d0e264ce45951a25b28ff129b187)
2019-04-04 11:45:20 +02:00
Dimitris Athanasiou 65cca2ee6f
[7.x][ML] Scrolling datafeed should clear scroll contexts on error (#40773) (#40794)
Closes #40772
2019-04-04 12:28:06 +03:00
Alexander Reelsen 6ac307d468 Replace javax activation with jakarta activation (#40247)
The eclipse foundation has taken over the javax.activation dependency, which
resulted in a naming change of the dependency.
2019-04-04 10:28:24 +02:00
Tim Vernum 1a30ab22fb
Show SSL usage when security is not disabled (#40761)
It is possible to have SSL enabled but security disabled if security
was dynamically disabled by the license type (e.g. trial license).

e.g. In the following configuration:

    xpack.license.self_generated.type: trial
    # xpack.security not set, default to disabled on trial
    xpack.security.transport.ssl.enabled: true

The security feature will be reported as

    available: true
    enabled: false

And in this case, SSL will be active even though security is not
enabled.

This commit causes the X-Pack feature usage to report the state of the
"ssl" features unless security was explicitly disabled in the
settings.

Backport of: #40672
2019-04-04 14:40:15 +11:00
Marios Trivyzas 3844da318f
SQL: Fix deserialisation issue of TimeProcessor (#40776)
TimeProcessor didn't implement `getWriteableName()` so the one from
the parent was used which returned the `NAME` of the parent. This
caused `TimeProcessor` objects to be deserialised into
DateTimeProcessor.

Moreover, added a restriction to run the TIME related integration tests
only in UTC timezone.

Fixes: #40717

(cherry picked from commit cfea348bec20e547df72c415cccd85279accb767)
2019-04-03 16:42:56 +02:00
David Roberts 324fef0548
[TEST] Mute WatchMetadataTests.testWatchMetadata
Due to https://github.com/elastic/elasticsearch/issues/40631
2019-04-03 13:56:33 +02:00
Marios Trivyzas 952c4d9653
SQL: Fix display size for DATE/DATETIME (#40669)
A full format for a DATETIME would be:
`2019-03-30T10:20:30.123+10:00` which is 29 chars long.

For DATE a full format would be: `2019-03-30T00:00:00.000+10:00`
which is also 29 chars long.


(cherry picked from commit 6be83964ed025528778bca8d35692762e166983b)
2019-04-03 13:29:04 +02:00
David Turner 1d2bc85586 Inline TransportReplAction#registerRequestHandlers (#40762)
It is important that resync actions are not rejected on the primary even if its
`write` threadpool is overloaded. Today we do this by exposing
`registerRequestHandlers` to subclasses and overriding it in
`TransportResyncReplicationAction`. This isn't ideal because it obscures the
difference between this action and other replication actions, and also might
allow subclasses to try and use some state before they are properly
initialised. This change replaces this override with a constructor parameter to
solve these issues.

Relates #40706
2019-04-03 12:12:26 +01:00
Hendrik Muhs 31e79a73d7 add HLRC protocol tests for transform state and stats (#40766)
adds HLRC protocol tests for state and stats hrlc clients
2019-04-03 12:51:15 +02:00
Hendrik Muhs 1f947054ff add reason to DataFrameTransformState and add hlrc protocol tests (#40736)
add field "reason" to DataFrameTransformState, add hlrc protocol tests and allow unknown fields for DataFrameTransformState
2019-04-03 07:35:07 +02:00
Benjamin Trent 945e7ca01e
[ML] Periodically persist data-frame running statistics to internal index (#40650) (#40729)
* [ML] Add mappings, serialization, and hooks to persist stats

* Adding tests for transforms without tasks having stats persisted

* intermittent commit

* Adjusting usage stats to account for stored stats docs

* Adding tests for id expander

* Addressing PR comments

* removing unused import

* adding shard failures to the task response
2019-04-02 14:16:55 -05:00
Zachary Tong abbfc75052 Remove timezone validation on rollup range queries (#40647)
We enforced the timezone of range queries when using the rollup
search endpoint, but this validation is not needed.  Since
rollup dates are stored in UTC, and range queries are always
converted to UTC (even if specifying a `time_zone`) the validation
is not needed and can prevent legitimate queries from running.
2019-04-02 14:25:16 -04:00
Benjamin Trent 4842d7fb7d
[ML] addressing test failure (#40701) (#40728)
* [ML] Fixing test

* adjusting line lengths

* marking valid seqno as final
2019-04-02 12:33:51 -05:00
Benjamin Trent 29180cefac
[ML] fix test check as randomness allows for different hours (#40536) (#40727)
* [ML] fix test check as randomness allows for different hours

* Re-enabling test
2019-04-02 12:33:35 -05:00
Lee Hinman 2fd01cc0b7 Fix testRunStateChangePolicyWithAsyncActionNextStep race condition (#40707)
Previously we only set the latch countdown with `nextStep.setLatch` after the
cluster state change has already been counted down. However, it's possible
execution could have already started, causing the latch to be missed when the
`MockAsyncActionStep` is being executed.

This moves the latch setting to be before the call to
`runPolicyAfterStateChange`, which means it is always available when the
`MockAsyncActionStep` is executed.

I was able to reproduce the failure every 30-40 runs before this change. With
this change, running 2000+ times the test passes.

Resolves #40018
2019-04-02 10:56:44 -06:00
Marios Trivyzas b78d63d624
SQL: [TEST] Mute TIME related failing tests
Mute failing tests tracked by: #40717

(cherry picked from commit 23719bbd687546500f1b2d2b86069a7efbf5bd29)
2019-04-02 15:48:18 +02:00
Benjamin Trent 4772b5d636
[ML] Addressing bug streaming DatafeedConfig aggs from (<= 6.5.4) -> 6.7.0 (#40659)
* [ML] Addressing bug streaming DatafeedConfig aggs from (<= 6.5.4) -> 6.7.0 (#40610)

* Addressing stream failure and adding tests to catch such in the future

* Add aggs to full cluster restart tests

* Test BWC for datafeeds with and without aggs

The wire serialisation is different for null/non-null
aggs, so it's worth testing both cases.

* Fixing bwc test, removing types

* Fixing BWC test for datafeed

* Update 40_ml_datafeed_crud.yml

* Update build.gradle
2019-04-02 07:04:33 -05:00
Tim Vernum 2c770ba3cb
Support mustache templates in role mappings (#40571)
This adds a new `role_templates` field to role mappings that is an
alternative to the existing roles field.

These templates are evaluated at runtime to determine which roles should be
granted to a user.
For example, it is possible to specify:

    "role_templates": [
      { "template":{ "source": "_user_{{username}}" } }
    ]

which would mean that every user is assigned to their own role based on
their username.

You may not specify both roles and role_templates in the same role
mapping.

This commit adds support for templates to the role mapping API, the role
mapping engine, the Java high level rest client, and Elasticsearch
documentation.

Due to the lack of caching in our role mapping store, it is currently
inefficient to use a large number of templated role mappings. This will be
addressed in a future change.

Backport of: #39984, #40504
2019-04-02 20:55:10 +11:00
Tim Vernum 7bdd41399d
Support roles with application privileges against wildcard applications (#40675)
This commit introduces 2 changes to application privileges:

- The validation rules now accept a wildcard in the "suffix" of an application name.
  Wildcards were always accepted in the application name, but the "valid filename" check
  for the suffix incorrectly prevented the use of wildcards there.

- A role may now be defined against a wildcard application (e.g. kibana-*) and this will
  be correctly treated as granting the named privileges against all named applications.
  This does not allow wildcard application names in the body of a "has-privileges" check, but the
  "has-privileges" check can test concrete application names against roles with wildcards.

Backport of: #40398
2019-04-02 14:48:39 +11:00
Marios Trivyzas 899ed2bf81 SQL: Introduce SQL TIME data type (#39802)
Support ANSI SQL's TIME type by introductin a runtime-only
ES SQL time type.

Closes: #38174
(cherry picked from commit 046ccd4cf0a251b2a3ddff6b072ab539a6711900)
2019-04-01 23:57:27 +02:00
Jay Modi 12bf3b2025
Remove with(out)-system-key tests (#40547) (#40603)
This change removes the variants of the rolling upgrade and full
cluster restart tests that use or do not use a system key. These tests
were added during 5.x when the system key was still used for security
and now the system key is only used as the watcher encryption key so
duplicating rolling upgrade and full cluster restarts is not needed.

The change here removes the subprojects for testing these scenarios and
defaults to always run with the watcher sensitive values encrypted for
these tests.
2019-04-01 14:23:24 -06:00
Andrei Stefan dfe4c6c568 SQL: have LIKE/RLIKE use wildcard and regexp queries (#40628)
* Have LIKE and RLIKE only use term-level queries (wildcard and regexp respectively). They
are already working only with exact fields, thus be in-line with how
SQL works in general (what you index is what you search on).

(cherry picked from commit 1bba887d481b49db231a1442922f1813952dcc67)
2019-04-01 18:15:48 +03:00
Andrei Stefan f8d3d685e5 SQL: Documentation for LIKE and RLIKE operators (#40623)
(cherry picked from commit 9536c5f7b7fecda4592f6b169adef180eadbba54)
2019-04-01 18:15:47 +03:00
Benjamin Trent 655e3d8f75
[ML] fix test, should account for async nature of audit (#40637) (#40683) 2019-04-01 10:00:32 -05:00
David Roberts 54b0ff8e33 [TEST] Mute WebhookHttpsIntegrationTests.testHttps
Due to https://github.com/elastic/elasticsearch/issues/35503
2019-04-01 14:20:01 +01:00
James Rodewig 39756fa538
[DOCS] Correct keystore commands for Email and Jira actions in Watcher (#40417) (#40612) 2019-04-01 08:25:26 -04:00
Marios Trivyzas 1b95ddfa3b
SQL: [Tests] Enable integration tests for fixed issues (#40664)
Enable some Ignored integration tests for issues/features that
have already been resolved/implemented.

(cherry picked from commit c23580f477ffc61c5701e14a91006db7bf21a8d4)
2019-03-30 16:57:19 +01:00
Marios Trivyzas 862fe612e0
SQL: Fix precedence of `::` psql like CAST operator (#40665)
Previously, an expression like `10 + 2::long` would be interpreted
as `CAST(10 + 2 AS LONG)` instead of `10 + CAST(2 AS LONG)`.


(cherry picked from commit e34cc2f38b1477e78788ee377938f42cc47187c7)
2019-03-30 12:51:40 +01:00
Marios Trivyzas 31776a4b15
SQL: [Docs] Fix example for CURDATE
Follows: 3dd0384d683940871f5d061f153b70c0420150fa
(cherry picked from commit 35d731754455ec91e20190a874592a740c66aa44)
2019-03-30 12:44:37 +01:00
Marios Trivyzas 6edb802716
SQL: [Docs] Fix doc errors regarding CURRENT_DATE. (#40649)
Some parts wrongly refered to CURRENT_TIMESTAMP.

(cherry picked from commit 3dd0384d683940871f5d061f153b70c0420150fa)
2019-03-30 12:08:38 +01:00
Ryan Ernst a0e3af80ff disable kerberos test until kerberos fixture is working again 2019-03-29 13:39:15 -07:00
krisds 336c92e4d3 [Contradictory type checks](https://lgtm.com/projects/g/elastic/elasticsearch/snapshot/dist-1916470085-1548143539391/files/x-pack/plugin/sql/jdbc/src/main/java/org/elasticsearch/xpack/sql/jdbc/debug/Debug.java#xe223478b74721d35:1) (#38154) 2019-03-29 17:01:14 +01:00
krisds 05ed1cfb33 Remove unused variable in QueryContainer 2019-03-29 17:01:14 +01:00
Jason Tedor 585f38787c
Add usage indicators for the bundled JDK (#40616)
This commit adds indications whether or not a distribution is from the
bundled JDK, and whether or not we are using the bundled JDK.
2019-03-29 08:25:32 -04:00
Ioannis Kakavas 8aca08bf52 Disable integTest when Docker is not available (#40585)
* Disable integTest when Docker is not available

Resolves: #40549
2019-03-29 08:15:01 +02:00
Christoph Büscher a13be65b01 Fixing typo in test error message (#40611) 2019-03-28 22:12:24 +01:00
Gordon Brown db7f00098e
Correct ILM metadata minimum compatibility version (#40569)
The ILM metadata minimum compatibility version was not set correctly,
which can cause issues in mixed-version clusters.
2019-03-28 10:53:44 -06:00
Lee Hinman d1357147f4 Handle null retention leases in WaitForNoFollowersStep (#40477)
In some cases the retention leases can return null, causing a
`NullPointerException` when waiting for no followers.

This wraps those so that no NPE is thrown.

Here is an example failure:

```
[2019-03-26T09:24:01,368][ERROR][o.e.x.i.IndexLifecycleRunner] [node-0] policy [deletePolicy] for index [ilm-00001] failed on step [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}]. Moving to ERROR step
java.lang.NullPointerException: null
	at org.elasticsearch.xpack.core.indexlifecycle.WaitForNoFollowersStep.lambda$evaluateCondition$0(WaitForNoFollowersStep.java:60) ~[?:?]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_191]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_191]
	at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958) ~[?:1.8.0_191]
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) ~[?:1.8.0_191]
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) ~[?:1.8.0_191]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) ~[?:1.8.0_191]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_191]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) ~[?:1.8.0_191]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) ~[?:1.8.0_191]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_191]
	at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:449) ~[?:1.8.0_191]
	at org.elasticsearch.xpack.core.indexlifecycle.WaitForNoFollowersStep.lambda$evaluateCondition$2(WaitForNoFollowersStep.java:61) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:383) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onNodeResponse(TransportBroadcastByNodeAction.java:352) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:324) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:314) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1095) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1176) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
...
```
2019-03-28 10:43:10 -06:00
Luca Cavanna 873c5638e6 Mute DataFrameAuditorIT#testAuditorWritesAudits
Relates to #40594
2019-03-28 16:53:54 +01:00
Alpar Torok d791e08932 Test fixtures krb5 (#40297)
Replaces the vagrant based kerberos fixtures with docker based test fixtures plugin.
The configuration is now entirely static on the docker side and no longer driven by Gradle,
also two different services are being configured since there are two different consumers of the fixture that can run in parallel and require different configurations.
2019-03-28 17:26:58 +02:00