Commit Graph

2372 Commits

Author SHA1 Message Date
Jake Landis 587034dfa7
Add set_priority action to ILM (#37397)
This commit adds a set_priority action to the hot, warm, and cold
phases for an ILM policy. This action sets the `index.priority`
on the managed index to allow different priorities between the
hot, warm, and cold recoveries.

This commit also includes the HLRC and documentation changes.

closes #36905
2019-01-17 09:55:36 -06:00
Martijn van Groningen b85bfd3e17
Added fatal_exception field for ccr stats in monitoring mapping. (#37563) 2019-01-17 14:04:41 +01:00
Martijn van Groningen 99b09845da
Moved ccr integration to the package with other ccr integration tests. 2019-01-17 13:57:56 +01:00
Marios Trivyzas 1686c32ba9
SQL: Rename SQL type DATE to DATETIME (#37395)
* SQL: Rename SQL data type DATE to DATETIME

SQL data type DATE has only the date part (e.g.: 2019-01-14)
without any time information. Previously the SQL type DATE was
referring to the ES DATE which contains also the time part along
with TZ information. To conform with SQL data types the data type
`DATE` is renamed to `DATETIME`, since it includes also the time,
as a new runtime SQL `DATE` data type will be introduced down the road,
which only contains the date part and meets the SQL standard.

Closes: #36440

* Address comments
2019-01-17 10:17:58 +02:00
Marios Trivyzas ecf0de30ba
SQL: Lowercase the datatypes in validation error msgs (#37524)
To follow the ES convention display the datatypes in lowercase
in error messages thrown during validation if `IN` and conditional
functions.
2019-01-16 18:41:10 +02:00
Marios Trivyzas 2cf4b1863f
SQL: Lowercase es data type (mapping) returned from SQL Commands (#37531)
To follow the ES convention, convert the es data type, returned as
column `mapping` from SQL Commands, to lowercase.

Fixes: #37521
2019-01-16 18:08:33 +02:00
Costin Leau 19d2c29ca6 Remove SYS CATALOGS leftover
Relates #37506
2019-01-16 17:28:06 +02:00
Costin Leau 1f76b5fc31
SQL: Describe aliases as views (#37496)
When reporting metadata, several clients have issues with the 'ALIAS'
type. To improve compatibility and be consistent with the ANSI SQL
expectations and because they are similar, aliases targets are now
reported as views.

Close #37422
2019-01-16 17:26:00 +02:00
Tim Brooks 0b5af276a8
Allow system privilege to execute proxied actions (#37508)
Currently all proxied actions are denied for the `SystemPrivilege`.
Unfortunately, there are use cases (CCR) where we would like to proxy
actions to a remote node that are normally performed by the
system context. This commit allows the system context to perform
proxy actions if they are actions that the system context is normally
allowed to execute.
2019-01-16 07:52:38 -07:00
Andrei Stefan 659326fdd6
SQL: Add protocol tests and remove jdbc_type from drivers response (#37516) 2019-01-16 16:28:46 +02:00
David Kyle 75410dc632 [Ml] Prevent config snapshot failure blocking migration (#37493) 2019-01-16 11:51:15 +00:00
Costin Leau 023bb2f1e4
SQL: Remove slightly used meta commands (#37506)
Remove SYS CATALOGS and SYS TABLE TYPES as they are a subset of SYS
TABLES (and thus somewhat redundant) and used only by JDBC.

Close #37409
2019-01-16 12:36:35 +02:00
Przemyslaw Gomulka 5e94f384c4
Remove the use of AbstracLifecycleComponent constructor #37488 (#37488)
The AbstracLifecycleComponent used to extend AbstractComponent, so it had to pass settings to the constractor of its supper class.
It no longer extends the AbstractComponent so there is no need for this constructor
There is also no need for AbstracLifecycleComponent subclasses to have Settings in their constructors if they were only passing it over to super constructor.
This is part 1. which will be backported to 6.x with a migration guide/deprecation log.
part 2 will have this constructor removed in 7
relates #35560

relates #34488
2019-01-16 09:05:30 +01:00
Hendrik Muhs 15d1b904a1
[ML] log minimum diskspace setting if forecast fails due to insufficient d… (#37486)
log minimum disk space setting if forecast fails due to insufficient disk space
2019-01-16 08:10:13 +01:00
Jay Modi 987576b013
Consistently use loopback address for ssl profile (#37487)
This change fixes failures in the SslMultiPortTests where we attempt to
connect to a profile on a port it is listening on but the connection
fails. The failure is due to the profile being bound to multiple
addresses and randomization will pick one of these addresses to
determine the listening port. However, the address we get the port for
may not be the address we are actually connecting to. In order to
resolve this, the test now sets the bind host for profiles to the
loopback address and uses the same address for connecting.

Closes #37481
2019-01-15 14:03:21 -07:00
David Kyle bea46f7b52
[ML] Migrate unallocated jobs and datafeeds (#37430)
Migrate ml job and datafeed config of open jobs and update
the parameters of the persistent tasks as they become unallocated
during a rolling upgrade. Block allocation of ml persistent tasks
until the configs are migrated.
2019-01-15 18:21:39 +00:00
David Kyle 7c11b05c28
[ML] Remove unused code from the JIndex project (#37477) 2019-01-15 17:19:58 +00:00
Marios Trivyzas 6129e9d9dd Revert "[TEST] Muted TokenBackwardsCompatibilityIT.*"
This reverts commit 65e42ab63b.

The test is only failing in 6.x not master.
2019-01-15 18:25:53 +02:00
Marios Trivyzas 65e42ab63b [TEST] Muted TokenBackwardsCompatibilityIT.*
Relates to #37379
2019-01-15 18:05:20 +02:00
Martijn van Groningen 9554b2fecb
When removing an AutoFollower also mark it as removed. (#37402)
Currently when there are no more auto follow patterns for a remote cluster then
the AutoFollower instance for this remote cluster will be removed. If
a new auto follow pattern for this remote cluster gets added quickly enough
after the last delete then there may be two AutoFollower instance running
for this remote cluster instead of one.

Each AutoFollower instance stops automatically after it sees in the
start() method that there are no more auto follow patterns for the
remote cluster it is tracking. However when an auto follow pattern
gets removed and then added back quickly enough then old AutoFollower
may never detect that at some point there were no auto follow patterns
for the remote cluster it is monitoring. The creation and removal of
an AutoFollower instance happens independently in the `updateAutoFollowers()`
as part of a cluster state update.

By adding the `removed` field, an AutoFollower instance will not miss the
fact there were no auto follow patterns at some point in time. The
`updateAutoFollowers()` method now marks an AutoFollower instance as
removed when it sees that there are no more patterns for a remote cluster.
The updateAutoFollowers() method can then safely start a new AutoFollower
instance.

Relates to #36761
2019-01-15 16:24:19 +01:00
Jay Modi a56aa4f076
Remove SslNullCipherTests from codebase (#37431)
This change deletes the SslNullCipherTests from our codebase since it
will have issues with newer JDK versions and it is essentially testing
JDK functionality rather than our own. The upstream JDK issue for
disabling these ciphers by default is
https://bugs.openjdk.java.net/browse/JDK-8212823.

Closes #37403
2019-01-15 07:52:58 -07:00
David Roberts 7cdf7f882b
[ML] Fix ML datafeed CCS with wildcarded cluster name (#37470)
The test that remote clusters used by ML datafeeds have
a license that allows ML was not accounting for the
possibility that the remote cluster name could be
wildcarded.  This change fixes that omission.

Fixes #36228
2019-01-15 14:19:05 +00:00
Tanguy Leroux e848388865
Fix SourceOnlySnapshotIT (#37461)
The SourceOnlySnapshotIT class tests a source only repository
using the following scenario:
    starts a master node
    starts a data node
    creates a source only repository
    creates an index with documents
    snapshots the index to the source only repository
    deletes the index
    stops the data node
    starts a new data node
    restores the index

Thanks to ESIntegTestCase the index is sometimes created using a custom 
data path. With such a setting, when a shard is assigned to one of the data 
node of the cluster the shard path is resolved using the index custom data 
path and the node's lock id by the NodeEnvironment#resolveCustomLocation().

It should work nicely but in SourceOnlySnapshotIT.snashotAndRestore(), b
efore the change in this PR, the last data node was restarted using a different 
path.home. At startup time this node was assigned a node lock based on other 
locks in the data directory of this temporary path.home which is empty. So it 
always got the 0 lock id. And when this new data node is assigned a shard for
 the index and resolves it against the index custom data path, it also uses the 
node lock id 0 which conflicts with another node of the cluster, resulting in 
various errors with the most obvious one being LockObtainFailedException.

This commit removes the temporary home path for the last data node so that it 
uses the same path home as other nodes of the cluster and then got assigned 
a correct node lock id at startup.

Closes #36330
Closes #36276
2019-01-15 15:03:09 +01:00
Albert Zaharovits a88c050a05
Docs be explicit on how to turn off deprecated auditing (#37316)
Just be explicit about turning off the deprecated audit log appender
because we really want people to turn it off.
2019-01-15 14:29:32 +02:00
Marios Trivyzas b594e81c86
SQL: Fix issue with field names containing "." (#37364)
Adjust FieldExtractor to handle fields which contain `.` in their
name, regardless where they fall in, in the document hierarchy. E.g.:

```
{
  "a.b": "Elastic Search"
}

{
  "a": {
    "b.c": "Elastic Search"
  }
}

{
  "a.b": {
    "c": {
      "d.e" : "Elastic Search"
    }
  }
}
```

Fixes: #37128
2019-01-15 09:41:41 +02:00
Jason Tedor 3bc0711b90
Add simple method to write collection of writeables (#37448)
This commit adds a simple convenience method for writing a collection of
writeables, and replaces existing call sites with the new method.
2019-01-14 21:28:28 -05:00
Julie Tibshirani 36a3b84fc9
Update the default for include_type_name to false. (#37285)
* Default include_type_name to false for get and put mappings.

* Default include_type_name to false for get field mappings.

* Add a constant for the default include_type_name value.

* Default include_type_name to false for get and put index templates.

* Default include_type_name to false for create index.

* Update create index calls in REST documentation to use include_type_name=true.

* Some minor clean-ups around the get index API.

* In REST tests, use include_type_name=true by default for index creation.

* Make sure to use 'expression == false'.

* Clarify the different IndexTemplateMetaData toXContent methods.

* Fix FullClusterRestartIT#testSnapshotRestore.

* Fix the ml_anomalies_default_mappings test.

* Fix GetFieldMappingsResponseTests and GetIndexTemplateResponseTests.

We make sure to specify include_type_name=true during xContent parsing,
so we continue to test the legacy typed responses. XContent generation
for the typeless responses is currently only covered by REST tests,
but we will be adding unit test coverage for these as we implement
each typeless API in the Java HLRC.

This commit also refactors GetMappingsResponse to follow the same appraoch
as the other mappings-related responses, where we read include_type_name
out of the xContent params, instead of creating a second toXContent method.
This gives better consistency in the response parsing code.

* Fix more REST tests.

* Improve some wording in the create index documentation.

* Add a note about types removal in the create index docs.

* Fix SmokeTestMonitoringWithSecurityIT#testHTTPExporterWithSSL.

* Make sure to mention include_type_name in the REST docs for affected APIs.

* Make sure to use 'expression == false' in FullClusterRestartIT.

* Mention include_type_name in the REST templates docs.
2019-01-14 13:08:01 -08:00
Jay Modi f3edbe2911
Security: remove SSL settings fallback (#36846)
This commit removes the fallback for SSL settings. While this may be
seen as a non user friendly change, the intention behind this change
is to simplify the reasoning needed to understand what is actually
being used for a given SSL configuration. Each configuration now needs
to be explicitly specified as there is no global configuration or
fallback to some other configuration.

Closes #29797
2019-01-14 14:06:22 -07:00
Shaunak Kashyap b86621c157
Adding mapping for hostname field (#37288)
This new `hostname` field is meant to be a replacement for its sibling `name` field. See https://github.com/elastic/beats/pull/9943, particularly https://github.com/elastic/beats/pull/9943#discussion_r245932581.

This PR simply adds the new field (`hostname`) to the mapping without removing the old one (`name`), because a user might be running an older-version Beat (without this field rename in it) with a newer-version Monitoring ES cluster (with this PR's change in it).

AFAICT the Monitoring UI isn't currently using the `name` field so no changes are necessary there yet. If it decides to start using the `name` field, it will also want to look at the value of the `hostname` field.
2019-01-14 12:41:10 -08:00
Tim Brooks 5c68338a1c
Implement ccr file restore (#37130)
This is related to #35975. It implements a file based restore in the
CcrRepository. The restore transfers files from the leader cluster
to the follower cluster. It does not implement any advanced resiliency
features at the moment. Any request failure will end the restore.
2019-01-14 13:07:55 -07:00
David Kyle 2ee55a50bf
[ML] Use String rep of Version in map for serialisation (#37416) 2019-01-14 16:39:47 +00:00
Martijn van Groningen de852765d6
unmuted test
Relates to #37014
2019-01-14 14:27:42 +01:00
Ioannis Kakavas 374e24c7fd Mute SslNullCipherTests on JDK12
JDK12 doesn't support NULL cipher for TLS by default. This commit
mutes these tests on JDK12 until we decide whether we need to keep
or remove them
2019-01-14 10:50:24 +02:00
Alpar Torok a566bacbc8 Upgrade ASM for java 12 compatability (#37385)
Closes #37371
2019-01-13 09:33:39 -08:00
Albert Zaharovits 6fd57d90da
Security Audit includes HTTP method for requests (#37322)
Adds another field, named "request.method", to the structured logfile audit.
This field is present for all events associated with a REST request (not a
transport request) and the value is one of GET, POST, PUT, DELETE, OPTIONS,
HEAD, PATCH, TRACE and CONNECT.
2019-01-13 15:26:23 +02:00
Costin Leau a4339ec7e9
SQL: Use declared source for error messages (#37161)
Improve error messages by returning the original SQL statement
declaration instead of trying to reproduce it as the casing and
whitespaces are not preserved accurately leading to small 
differences.

Close #37161
2019-01-13 01:40:22 +02:00
Marios Trivyzas 359222c55c
SQL: Make `FULL` non-reserved keyword in the grammar (#37377)
Since `full` can be common as a field name or part of a field name
(e.g.: `full.name` or `name.full`), it's nice if it's not a reserved
keyword of the grammar so a user can use it without resorting to quotes.

Fixes: #37376
2019-01-11 23:08:00 +02:00
Marios Trivyzas 85531f0285
SQL: [Tests] Fix and enable internalClusterTests (#37300)
SqlPlugin cannot have more than one public constructor, so for the testing
purposes the `getLicenseState()` should be overriden.

Fixes: #37191

Co-authored-by: Michael Basnight <mbasnight@gmail.com>
2019-01-11 22:43:17 +02:00
Benjamin Trent 5101e51891
ML: Fix testMigrateConfigs (#37373)
* ML: :s/execute/get

* Fixing other broken tests

* unmuting test
2019-01-11 13:29:30 -06:00
Zachary Tong de52ba1f78 Fix RollupDocumentation test to wait for job to stop
Also adds some extra state debug information to various log messages
2019-01-11 14:14:58 -05:00
Gordon Brown 827ece73c8 Mute MlConfigMigratorIT.testMigrateConfigs (#37374) 2019-01-11 11:11:58 -07:00
Gordon Brown 955d3aea19 Mute testRoundRobinWithFailures (#32190) 2019-01-11 09:38:40 -07:00
David Roberts 953fb9352f
[ML] Update error message for process update (#37363)
When this message was first added the model debug config was
the only thing that could be updated, but now more aspects of
the config can be updated so the message needs to be more
general.
2019-01-11 16:31:55 +00:00
Martijn van Groningen e4391afd98
Test fix, wait for auto follower to have stopped in the background
Relates to #36761
2019-01-11 17:26:17 +01:00
Benjamin Trent 19a7e0f4eb
ML: update .ml-state actions to support > 1 index (#37307)
* ML: Updating .ml-state calls to be able to support > 1 index

* Matching bulk delete behavior with dbq

* Adjusting state name

* refreshing indices before search

* fixing line length

* adjusting index expansion options
2019-01-11 08:03:41 -06:00
David Roberts 1da59db3fb
[ML] Wait for autodetect to be ready in the datafeed (#37349)
This is a reinforcement of #37227.  It turns out that
persistent tasks are not made stale if the node they
were running on is restarted and the master node does
not notice this.  The main scenario where this happens
is when minimum master nodes is the same as the number
of nodes in the cluster, so the cluster cannot elect a
master node when any node is restarted.

When an ML node restarts we need the datafeeds for any
jobs that were running on that node to not just wait
until the jobs are allocated, but to wait for the
autodetect process of the job to start up.  In the case
of reassignment of the job persistent task this was
dealt with by the stale status test.  But in the case
where a node restarts but its persistent tasks are not
reassigned we need a deeper test.

Fixes #36810
2019-01-11 13:22:35 +00:00
Alexander Reelsen bbd093059f
Add whitelist to watcher HttpClient (#36817)
This adds a configurable whitelist to the HTTP client in watcher. By
default every URL is allowed to retain BWC. A dynamically configurable
setting named "xpack.http.whitelist" was added that allows to
configure an array of URLs, which can also contain simple regexes.

Closes #29937
2019-01-11 09:22:47 +01:00
Martijn van Groningen 37493c204d
Unmuted test now that #37239 has been merged and backported.
Relates to #37231
2019-01-11 09:02:46 +01:00
Ioannis Kakavas 80084138dd [DOCS] Fix link to role mapping doc 2019-01-11 09:22:40 +02:00
markharwood 434430506b
Type removal - added deprecation warnings to _bulk apis (#36549)
Added warnings checks to existing tests
Added “defaultTypeIfNull” to DocWriteRequest interface so that Bulk requests can override a null choice of document type with any global custom choice.
Related to #35190
2019-01-10 21:35:19 +00:00