Commit Graph

2971 Commits

Author SHA1 Message Date
Marios Trivyzas ddf17dfb1e
SQL: Small code improvements of Pipes & Processors (#40909)
- Remove superfluous methods that are already
defined in superclasses.
- Improve tests for null folding on conditionals

(cherry picked from commit 67f9404f5004362e569353d1e950ffe5d7a9ab6e)
2019-04-08 10:14:13 +02:00
Tim Vernum 26c63e0115
Add test for HTTP and Transport TLS on basic license (#40932)
This adds a new security/qa test for TLS on a basic license.

It starts a 2 node cluster with a basic license, and TLS enabled
on both HTTP and Transport, and verifies the license type, x-pack
SSL usage and SSL certificates API.

It also upgrades the cluster to a trial license and performs that
same set of checks (to ensure that clusters with basic license
and TLS enabled can be upgraded to a higher feature license)

Backport of: #40714
2019-04-08 13:23:12 +10:00
Hendrik Muhs d5fcbf2f4a refactor onStart and onFinish to take runnables and executed them guarded by state (#40855)
refactor onStart and onFinish to take action listeners and execute them when indexer is in indexing state.
2019-04-07 21:46:37 +02:00
Jay Modi f34663282c
Update apache httpclient to version 4.5.8 (#40875)
This change updates our version of httpclient to version 4.5.8, which
contains the fix for HTTPCLIENT-1968, which is a bug where the client
started re-writing paths that contained encoded reserved characters
with their unreserved form.
2019-04-05 13:48:10 -06:00
David Turner 2ff19bc1b7
Use Writeable for TransportReplAction derivatives (#40905)
Relates #34389, backport of #40894.
2019-04-05 19:10:10 +01:00
Benjamin Trent a8dbb07546
[ML] Changes default destination index field mapping and adds scripted_metric agg (#40750) (#40846)
* [ML] Allowing destination index mappings to have dynamic types, adds script_metric agg

* Making dynamic|source mapping explicit
2019-04-05 11:34:20 -05:00
Brandon Kobel eb656244bf Add Kibana application privileges for monitoring and ml reserved roles (#40651)
* Add Kibana application privileges for monitoring and ml reserved roles

* Adding test for kibana-.kibana application explicitly

* Whoa there, fat fingered kibana and application...

* And I copied something from monitoring I shouldn't have...

* And actually doing what Yogesh recommended...
2019-04-05 07:44:29 -07:00
Benjamin Trent 665f0d81aa
[ML] refactoring start task a bit, removing unused code (#40798) (#40845) 2019-04-05 09:01:01 -05:00
Tim Vernum 920024e06d
Remove dynamic objects from security index (#40715)
The security index had a few "object" types with

   "dynamic": true

However, this automatically creates a mapping for each field that is
created within those objects. This means that types are dynamically
inferred and "locked in" for future updates.

Instead we want "dynamic": false which will allow us to store a range
of fields in these nested objects and retrieve them from the source,
without creating mapping types for those fields.

Backport of: #40499
2019-04-05 18:28:09 +11:00
Martijn van Groningen 809a5f13a4
Make -try xlint warning disabled by default. (#40833)
Many gradle projects specifically use the -try exclude flag, because
there are many cases where auto-closeable resource ignore is never
referenced in body of corresponding try statement. Suppressing this
warning specifically in each case that it happens using
`@SuppressWarnings("try")` would be very verbose.

This change removes `-try` from any gradle project and adds it to the
build plugin. Also this change removes exclude flags from gradle projects
that is already specified in build plugin (for example -deprecation).

Relates to #40366
2019-04-05 08:02:26 +02:00
Jim Ferenczi a15f55b2de Rollup ignores time_zone on date histogram (#40844)
When translating the original aggregation for the rollup indices,
the timezone of the date histogram is validated against the rollup
job but the value is not copied in the newly created date_histogram.
2019-04-04 21:16:50 +02:00
Gordon Brown 5347dec55e
Allow ILM to stop if indices have nonexistent policies (#40820)
Prior to this PR, there is a bug in ILM which does not allow ILM to stop
if one or more indices have an index.lifecycle.name which refers to
a policy that does not exist - the operation_mode will be stuck as
STOPPING until either the policy is created or the nonexistent
policy is removed from those indices.

This change allows ILM to stop in this case and makes the logging more
clear as to why ILM is not stopping.
2019-04-04 11:46:21 -06:00
Tim Vernum 9d34164ce6
Add build utility to check cluster health over ssl (#40713)
By default, in integ tests we wait for the standalone cluster to start
by using the ant Get task to retrieve the cluster health endpoint.
However the ant task has no facilities for customising the trusted
CAs for a https resource, so if the integ test cluster has TLS enabled
on the http interface (using a custom CA) we need a separate utility
for that purpose.

Backport of: #40573
2019-04-04 21:44:03 +11:00
David Roberts d16f86f7ab [ML] Add created_by info to usage stats (#40518)
This change adds information about which UI path
(if any) created ML anomaly detector jobs to the
stats returned by the _xpack/usage endpoint.

Counts for the following possibilities are expected:

* ml_module_apache_access
* ml_module_apm_transaction
* ml_module_auditbeat_process_docker
* ml_module_auditbeat_process_hosts
* ml_module_nginx_access
* ml_module_sample
* multi_metric_wizard
* population_wizard
* single_metric_wizard
* unknown

The "unknown" count is for jobs that do not have a
created_by setting in their custom_settings.

Closes #38403
2019-04-04 10:55:20 +01:00
Marios Trivyzas f3c207d27b SQL: Implement CURRENT_TIME/CURTIME functions (#40662)
After `TIME` SQL data type is introduced, implement
`CURRENT_TIME/CURTIME` functions similarly to CURRENT_TIMESTAMP
that return the system's current time (only, without the date part).

Closes: #40468
(cherry picked from commit 9feede781409d0e264ce45951a25b28ff129b187)
2019-04-04 11:45:20 +02:00
Dimitris Athanasiou 65cca2ee6f
[7.x][ML] Scrolling datafeed should clear scroll contexts on error (#40773) (#40794)
Closes #40772
2019-04-04 12:28:06 +03:00
Alexander Reelsen 6ac307d468 Replace javax activation with jakarta activation (#40247)
The eclipse foundation has taken over the javax.activation dependency, which
resulted in a naming change of the dependency.
2019-04-04 10:28:24 +02:00
Tim Vernum 1a30ab22fb
Show SSL usage when security is not disabled (#40761)
It is possible to have SSL enabled but security disabled if security
was dynamically disabled by the license type (e.g. trial license).

e.g. In the following configuration:

    xpack.license.self_generated.type: trial
    # xpack.security not set, default to disabled on trial
    xpack.security.transport.ssl.enabled: true

The security feature will be reported as

    available: true
    enabled: false

And in this case, SSL will be active even though security is not
enabled.

This commit causes the X-Pack feature usage to report the state of the
"ssl" features unless security was explicitly disabled in the
settings.

Backport of: #40672
2019-04-04 14:40:15 +11:00
Marios Trivyzas 3844da318f
SQL: Fix deserialisation issue of TimeProcessor (#40776)
TimeProcessor didn't implement `getWriteableName()` so the one from
the parent was used which returned the `NAME` of the parent. This
caused `TimeProcessor` objects to be deserialised into
DateTimeProcessor.

Moreover, added a restriction to run the TIME related integration tests
only in UTC timezone.

Fixes: #40717

(cherry picked from commit cfea348bec20e547df72c415cccd85279accb767)
2019-04-03 16:42:56 +02:00
David Roberts 324fef0548
[TEST] Mute WatchMetadataTests.testWatchMetadata
Due to https://github.com/elastic/elasticsearch/issues/40631
2019-04-03 13:56:33 +02:00
Marios Trivyzas 952c4d9653
SQL: Fix display size for DATE/DATETIME (#40669)
A full format for a DATETIME would be:
`2019-03-30T10:20:30.123+10:00` which is 29 chars long.

For DATE a full format would be: `2019-03-30T00:00:00.000+10:00`
which is also 29 chars long.


(cherry picked from commit 6be83964ed025528778bca8d35692762e166983b)
2019-04-03 13:29:04 +02:00
David Turner 1d2bc85586 Inline TransportReplAction#registerRequestHandlers (#40762)
It is important that resync actions are not rejected on the primary even if its
`write` threadpool is overloaded. Today we do this by exposing
`registerRequestHandlers` to subclasses and overriding it in
`TransportResyncReplicationAction`. This isn't ideal because it obscures the
difference between this action and other replication actions, and also might
allow subclasses to try and use some state before they are properly
initialised. This change replaces this override with a constructor parameter to
solve these issues.

Relates #40706
2019-04-03 12:12:26 +01:00
Hendrik Muhs 31e79a73d7 add HLRC protocol tests for transform state and stats (#40766)
adds HLRC protocol tests for state and stats hrlc clients
2019-04-03 12:51:15 +02:00
Hendrik Muhs 1f947054ff add reason to DataFrameTransformState and add hlrc protocol tests (#40736)
add field "reason" to DataFrameTransformState, add hlrc protocol tests and allow unknown fields for DataFrameTransformState
2019-04-03 07:35:07 +02:00
Benjamin Trent 945e7ca01e
[ML] Periodically persist data-frame running statistics to internal index (#40650) (#40729)
* [ML] Add mappings, serialization, and hooks to persist stats

* Adding tests for transforms without tasks having stats persisted

* intermittent commit

* Adjusting usage stats to account for stored stats docs

* Adding tests for id expander

* Addressing PR comments

* removing unused import

* adding shard failures to the task response
2019-04-02 14:16:55 -05:00
Zachary Tong abbfc75052 Remove timezone validation on rollup range queries (#40647)
We enforced the timezone of range queries when using the rollup
search endpoint, but this validation is not needed.  Since
rollup dates are stored in UTC, and range queries are always
converted to UTC (even if specifying a `time_zone`) the validation
is not needed and can prevent legitimate queries from running.
2019-04-02 14:25:16 -04:00
Benjamin Trent 4842d7fb7d
[ML] addressing test failure (#40701) (#40728)
* [ML] Fixing test

* adjusting line lengths

* marking valid seqno as final
2019-04-02 12:33:51 -05:00
Benjamin Trent 29180cefac
[ML] fix test check as randomness allows for different hours (#40536) (#40727)
* [ML] fix test check as randomness allows for different hours

* Re-enabling test
2019-04-02 12:33:35 -05:00
Lee Hinman 2fd01cc0b7 Fix testRunStateChangePolicyWithAsyncActionNextStep race condition (#40707)
Previously we only set the latch countdown with `nextStep.setLatch` after the
cluster state change has already been counted down. However, it's possible
execution could have already started, causing the latch to be missed when the
`MockAsyncActionStep` is being executed.

This moves the latch setting to be before the call to
`runPolicyAfterStateChange`, which means it is always available when the
`MockAsyncActionStep` is executed.

I was able to reproduce the failure every 30-40 runs before this change. With
this change, running 2000+ times the test passes.

Resolves #40018
2019-04-02 10:56:44 -06:00
Marios Trivyzas b78d63d624
SQL: [TEST] Mute TIME related failing tests
Mute failing tests tracked by: #40717

(cherry picked from commit 23719bbd687546500f1b2d2b86069a7efbf5bd29)
2019-04-02 15:48:18 +02:00
Benjamin Trent 4772b5d636
[ML] Addressing bug streaming DatafeedConfig aggs from (<= 6.5.4) -> 6.7.0 (#40659)
* [ML] Addressing bug streaming DatafeedConfig aggs from (<= 6.5.4) -> 6.7.0 (#40610)

* Addressing stream failure and adding tests to catch such in the future

* Add aggs to full cluster restart tests

* Test BWC for datafeeds with and without aggs

The wire serialisation is different for null/non-null
aggs, so it's worth testing both cases.

* Fixing bwc test, removing types

* Fixing BWC test for datafeed

* Update 40_ml_datafeed_crud.yml

* Update build.gradle
2019-04-02 07:04:33 -05:00
Tim Vernum 2c770ba3cb
Support mustache templates in role mappings (#40571)
This adds a new `role_templates` field to role mappings that is an
alternative to the existing roles field.

These templates are evaluated at runtime to determine which roles should be
granted to a user.
For example, it is possible to specify:

    "role_templates": [
      { "template":{ "source": "_user_{{username}}" } }
    ]

which would mean that every user is assigned to their own role based on
their username.

You may not specify both roles and role_templates in the same role
mapping.

This commit adds support for templates to the role mapping API, the role
mapping engine, the Java high level rest client, and Elasticsearch
documentation.

Due to the lack of caching in our role mapping store, it is currently
inefficient to use a large number of templated role mappings. This will be
addressed in a future change.

Backport of: #39984, #40504
2019-04-02 20:55:10 +11:00
Tim Vernum 7bdd41399d
Support roles with application privileges against wildcard applications (#40675)
This commit introduces 2 changes to application privileges:

- The validation rules now accept a wildcard in the "suffix" of an application name.
  Wildcards were always accepted in the application name, but the "valid filename" check
  for the suffix incorrectly prevented the use of wildcards there.

- A role may now be defined against a wildcard application (e.g. kibana-*) and this will
  be correctly treated as granting the named privileges against all named applications.
  This does not allow wildcard application names in the body of a "has-privileges" check, but the
  "has-privileges" check can test concrete application names against roles with wildcards.

Backport of: #40398
2019-04-02 14:48:39 +11:00
Marios Trivyzas 899ed2bf81 SQL: Introduce SQL TIME data type (#39802)
Support ANSI SQL's TIME type by introductin a runtime-only
ES SQL time type.

Closes: #38174
(cherry picked from commit 046ccd4cf0a251b2a3ddff6b072ab539a6711900)
2019-04-01 23:57:27 +02:00
Jay Modi 12bf3b2025
Remove with(out)-system-key tests (#40547) (#40603)
This change removes the variants of the rolling upgrade and full
cluster restart tests that use or do not use a system key. These tests
were added during 5.x when the system key was still used for security
and now the system key is only used as the watcher encryption key so
duplicating rolling upgrade and full cluster restarts is not needed.

The change here removes the subprojects for testing these scenarios and
defaults to always run with the watcher sensitive values encrypted for
these tests.
2019-04-01 14:23:24 -06:00
Andrei Stefan dfe4c6c568 SQL: have LIKE/RLIKE use wildcard and regexp queries (#40628)
* Have LIKE and RLIKE only use term-level queries (wildcard and regexp respectively). They
are already working only with exact fields, thus be in-line with how
SQL works in general (what you index is what you search on).

(cherry picked from commit 1bba887d481b49db231a1442922f1813952dcc67)
2019-04-01 18:15:48 +03:00
Andrei Stefan f8d3d685e5 SQL: Documentation for LIKE and RLIKE operators (#40623)
(cherry picked from commit 9536c5f7b7fecda4592f6b169adef180eadbba54)
2019-04-01 18:15:47 +03:00
Benjamin Trent 655e3d8f75
[ML] fix test, should account for async nature of audit (#40637) (#40683) 2019-04-01 10:00:32 -05:00
David Roberts 54b0ff8e33 [TEST] Mute WebhookHttpsIntegrationTests.testHttps
Due to https://github.com/elastic/elasticsearch/issues/35503
2019-04-01 14:20:01 +01:00
James Rodewig 39756fa538
[DOCS] Correct keystore commands for Email and Jira actions in Watcher (#40417) (#40612) 2019-04-01 08:25:26 -04:00
Marios Trivyzas 1b95ddfa3b
SQL: [Tests] Enable integration tests for fixed issues (#40664)
Enable some Ignored integration tests for issues/features that
have already been resolved/implemented.

(cherry picked from commit c23580f477ffc61c5701e14a91006db7bf21a8d4)
2019-03-30 16:57:19 +01:00
Marios Trivyzas 862fe612e0
SQL: Fix precedence of `::` psql like CAST operator (#40665)
Previously, an expression like `10 + 2::long` would be interpreted
as `CAST(10 + 2 AS LONG)` instead of `10 + CAST(2 AS LONG)`.


(cherry picked from commit e34cc2f38b1477e78788ee377938f42cc47187c7)
2019-03-30 12:51:40 +01:00
Marios Trivyzas 31776a4b15
SQL: [Docs] Fix example for CURDATE
Follows: 3dd0384d683940871f5d061f153b70c0420150fa
(cherry picked from commit 35d731754455ec91e20190a874592a740c66aa44)
2019-03-30 12:44:37 +01:00
Marios Trivyzas 6edb802716
SQL: [Docs] Fix doc errors regarding CURRENT_DATE. (#40649)
Some parts wrongly refered to CURRENT_TIMESTAMP.

(cherry picked from commit 3dd0384d683940871f5d061f153b70c0420150fa)
2019-03-30 12:08:38 +01:00
Ryan Ernst a0e3af80ff disable kerberos test until kerberos fixture is working again 2019-03-29 13:39:15 -07:00
krisds 336c92e4d3 [Contradictory type checks](https://lgtm.com/projects/g/elastic/elasticsearch/snapshot/dist-1916470085-1548143539391/files/x-pack/plugin/sql/jdbc/src/main/java/org/elasticsearch/xpack/sql/jdbc/debug/Debug.java#xe223478b74721d35:1) (#38154) 2019-03-29 17:01:14 +01:00
krisds 05ed1cfb33 Remove unused variable in QueryContainer 2019-03-29 17:01:14 +01:00
Jason Tedor 585f38787c
Add usage indicators for the bundled JDK (#40616)
This commit adds indications whether or not a distribution is from the
bundled JDK, and whether or not we are using the bundled JDK.
2019-03-29 08:25:32 -04:00
Ioannis Kakavas 8aca08bf52 Disable integTest when Docker is not available (#40585)
* Disable integTest when Docker is not available

Resolves: #40549
2019-03-29 08:15:01 +02:00
Christoph Büscher a13be65b01 Fixing typo in test error message (#40611) 2019-03-28 22:12:24 +01:00