Commit Graph

2359 Commits

Author SHA1 Message Date
Ryan Ernst 1a36046421
Core: Convert FormatDateTimeFormatter to DateFormatter (#36396)
This commit creates JodaDateFormatter to replace
FormatDateTimeFormatter. It converts all uses of the old class
to DateFormatter to allow a future change to use JavaDateFormatter
when appropriate.
2018-12-10 08:06:41 -08:00
Dimitris Athanasiou 891a37adaa
[ML] Use LinkedHashMap to preserve order in job and datafeed fields (#36389) 2018-12-10 15:05:47 +00:00
David Roberts 558f4ec278
Ignore zen2 discovery task in waitForPendingTasks (#36381)
Fixes #36380
2018-12-10 11:07:02 +00:00
Yannick Welsch 6e6e63d01d
Zen2: Move all mixed-version REST tests to Zen2 (#36398)
Moves all remaining (rolling-upgrade and mixed-version) REST tests to use Zen2. To avoid adding
extra configuration, it relies on Zen2 being set as the default discovery type. This required a few
smaller changes in other tests. I've removed AzureMinimumMasterNodesTests which tests Zen1
functionality and dates from a time where host providers were not configurable and each cloud
plugin had its own discovery.type, subclassing the ZenDiscovery class. I've also adapted a few tests
which were unnecessarily adding addTestZenDiscovery = false for the same legacy reasons. Finally,
this also moves the unconfigured-node-name REST test to Zen2, testing the auto-bootstrapping
functionality in development mode when no discovery configuration is provided.
2018-12-10 11:00:57 +01:00
Martijn van Groningen 633ab24017
[CCR] Restructured QA modules (#36404)
Renamed the follow qa modules:
`multi-cluster-downgraded-to-basic-license` to `downgraded-to-basic-license`
`multi-cluster-with-non-compliant-license` to `non-compliant-license`
`multi-cluster-with-security` to `security`

Moved the `chain` module into the `multi-cluster` module and
changed the `multi-cluster` to start 3 clusters.

Followup from #36031
2018-12-09 19:34:48 +01:00
Nhat Nguyen 95bafb0593 TEST: Always enable soft-deletes in ShardChangesTests 2018-12-09 02:57:13 -05:00
Michael Basnight b5b6e37a60
Deprecate X-Pack centric watcher endpoints (#36218)
This commit is part of our plan to deprecate and ultimately remove the use of
_xpack in the REST APIs.

Relates #35958
2018-12-08 12:57:16 -06:00
Ryan Ernst a27f2efca5
Core: Converge FormatDateTimeFormatter and DateFormatter apis (#36390)
This commit makes FormatDateTimeFormatter and DateFormatter apis close
to each other, so that the former can be removed in favor of the latter.
This PR does not change the uses of FormatDateTimeFormatter yet, so that
that future change can be purely mechanical.
2018-12-07 17:23:41 -08:00
Gordon Brown 85bba0c3ae
Simplify deprecation issue levels (#36326)
This commit gets rid of the 'NONE' and 'INFO' severity levels for
deprecation issues.

'NONE' is unused and does not make much sense as a severity level.
'INFO' can be separated into two categories: Either 1) we can
definitively tell there will be a problem with the cluster/node/index
configuration that can be resolved prior to upgrade, in which case
the issue should be a WARNING, or 2) we can't, because any issues would
be at the application level, for which the user should review the
deprecation logs and/or response headers.
2018-12-07 15:45:53 -07:00
Tim Brooks 8a53f2b464
Implement basic `CcrRepository` restore (#36287)
This is related to #35975. It implements a basic restore functionality
for the CcrRepository. When the restore process is kicked off, it
configures the new index as expected for a follower index. This means
that the index has a different uuid, the version is not incremented, and
the Ccr metadata is installed.

When the restore shard method is called, an empty shard is initialized.
2018-12-07 15:27:04 -07:00
David Roberts 9e8cfbb40d
[ML] Deprecate X-Pack centric ML endpoints (#36315)
This commit is part of our plan to deprecate and
ultimately remove the use of _xpack in the REST APIs.

Relates #35958
2018-12-07 20:34:11 +00:00
Dimitris Athanasiou 7870ae81b1
[ML] Prevent stack overflow while copying ML jobs and datafeeds (#36370)
ML jobs and datafeeds wrap collections into their unmodifiable
equivalents in their constructor. However, the copying builder
does not make a copy of some of those collections resulting
in wrapping those again and again. This can eventually result
to stack overflow.

This commit addressed this issue by copying the collections in
question in the copying builder constructor.

Closes #36360
2018-12-07 20:16:25 +00:00
Nik Everett ead2b9e08b
HLRC: Add rollup search (#36334)
Relates to #29827
2018-12-07 14:39:58 -05:00
Dimitris Athanasiou b8dba16376
[ML] Ensure total hits are tracked (#36374)
This is in preparation of the anticipated change
that will disable accurate total hits tracking in
searches.
2018-12-07 18:01:37 +00:00
Nhat Nguyen f2df0a5be4
Remove LocalCheckpointTracker#resetCheckpoint (#34667)
In #34474, we added a new assertion to ensure that the
LocalCheckpointTracker is always consistent with Lucene index. However,
we reset LocalCheckpoinTracker in testDedupByPrimaryTerm cause this
assertion to be violated.

This commit removes resetCheckpoint from LocalCheckpointTracker and
rewrites testDedupByPrimaryTerm without resetting the local checkpoint.

Relates #34474
2018-12-07 12:22:20 -05:00
Simon Willnauer 844ff80778 Harden FrozenEngineTests#testCircuitBreakerAccounting
This test tries to compare the CB stats from an InternalEngine
and a FrozenEngine but is subject to segement merges that might finish
and get committed after we read the breaker stats. This can cause
occational test failures.

Closes #36207
2018-12-07 15:38:16 +01:00
David Roberts 60289e7331 Mute SourceOnlySnapshotIT testSnapshotAndRestore
Due to https://github.com/elastic/elasticsearch/issues/36330
2018-12-07 09:49:55 +00:00
Dimitris Athanasiou 0dd73ef7da
[ML] Move consuming and closing results stream to result processor (#36314)
The results iterator is consuming and closing the results stream
once it is done. It seems this should not be the responsibility
of the results iterator. It stops the iterator from being reusable
for different processes where closing the stream is not desirable.
This commit is moving the consuming and closing of the results stream
into the autodetect result processor.
2018-12-07 09:33:51 +00:00
Alexander Reelsen 7693d538ca
Docs: Mention that fields with dots do not work (#36295)
The dot is used as a splitting character internally for looking up
values in the array compare condition, thus the user should use the
script condition in such cases.
2018-12-07 09:38:12 +01:00
Armin Braun e6d190613f
[ZEN2] Use Zen2 in REST Tests (#36300) 2018-12-07 09:15:11 +01:00
Nhat Nguyen 10feb75eb7
Upgrade to Lucene-8.0.0-snapshot-aaa64d70159 (#36335)
Includes:

LUCENE-8594: DV update are broken for updates on new field
LUCENE-8590: Optimize DocValues update datastructures
LUCENE-8593: Specialize single value numeric DV updates

Relates #36286
2018-12-06 20:33:25 -05:00
Tim Brooks 373c67dd7a
Add DirectByteBuffer strategy for transport-nio (#36289)
This is related to #27260. In Elasticsearch all of the messages that we
serialize to write to the network are composed of heap bytes. When you
read or write to a nio socket in java, the heap memory you passed down
must be copied to/from direct memory. The JVM internally does some
buffering of the direct memory, however it is essentially unbounded.

This commit introduces a simple mechanism of buffering and copying the
memory in transport-nio. Each network event loop is given a 64kb
DirectByteBuffer. When we go to read we use this buffer and copy the
data after the read. Additionally, when we go to write, we copy the data
to the direct memory before calling write. 64KB is chosen as this is the
default receive buffer size we use for transport-netty4
(NETTY_RECEIVE_PREDICTOR_SIZE).

Since we only have one buffer per thread, we could afford larger.
However, if we the buffer is large and not all of the data is flushed in
a write call, we will do excess copies. This is something we can
explore in the future.
2018-12-06 18:09:07 -07:00
Ryan Ernst 37b3fc383f
Build: Use explicit deps on test tasks for check (#36325)
This commit moves back to use explicit dependsOn for test tasks on
check. Not all tasks extending RandomizedTestingTask should be run by
check directly.
2018-12-06 14:13:49 -08:00
Benjamin Trent 3e04a90e99
[ML] Adding audits when deprecation warnings occur with datafeed start (#36233)
* [ML] Adding audits when deprecation warnings occur with datafeed start

* adjusting parameters for log format call
2018-12-06 15:58:37 -06:00
Julie Tibshirani 3f3cde41d3
Deprecate types in termvector and mtermvector requests. (#36182)
* Add deprecation warnings to `Rest*TermVectorsAction`, plus tests in `Rest*TermVectorsActionTests`.
* Deprecate relevant methods on the Java HLRC requests/ responses.
* Update documentation (for both the REST API and Java HLRC).
* For each REST yml test, create one version without types, and another legacy version that retains types (called *_with_types.yml).
2018-12-06 10:23:15 -08:00
Jason Tedor d4d3a3e467
Remove license state listeners on closables (#36308)
We have a few places where we register license state listeners on
transient components (i.e., resources that can be open and closed during
the lifecycle of the server). In one case (the opt-out query cache) we
were never removing the registered listener, effectively a terrible
memory leak. In another case, we were not un-registered the listener
that we registered, since we were not referencing the same instance of
Runnable. This commit does two things:
  - introduces a marker interface LicenseStateListener so that it is
    easier to identify these listeners in the codebase and avoid classes
    that need to register a license state listener from having to
    implement Runnable which carries a different semantic meaning than
    we want here
  - fixes the two places where we are currently leaking license state
    listeners
2018-12-06 12:52:04 -05:00
Benjamin Trent adc8355c5d
Adding additional tests for agg parsing in datafeedconfig (#36261)
* Adding additional tests for agg parsing in datafeedconfig

* Fixing bug, adding yml test
2018-12-06 11:19:34 -06:00
David Turner ed1c5a0241
Introduce `zen2` discovery type (#36298)
With this change it is now possible to start a node running Zen2.
2018-12-06 16:20:08 +00:00
Alexander Reelsen 97259f0efc
Docs: Add password keystore setting for email account passwords (#33409)
Always refer to secure settings when setting up email account passwords
2018-12-06 14:35:53 +01:00
Yannick Welsch ee05ef1312 Merge branch 'zen2' 2018-12-06 08:31:46 +01:00
Jim Ferenczi 3652f90552 Fix FullClusterRestartIT#testRollupIDSchemeAfterRestart
Set rest_total_hits_as_int in search requests only on the new cluster,
the old cluster can be on a version older than don't support this
parameter (< 6.6) and total hits is not an object in 6x anyway.

Closes #36291
2018-12-06 02:45:07 +01:00
Julie Tibshirani 331026b597 Mute FullClusterRestartIT#testRollupIDSchemeAfterRestart as we await a fix. 2018-12-05 17:05:28 -08:00
Yannick Welsch a0ae1cc987 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 23:13:12 +01:00
Andrey Ershov 5d6602120f [Zen2] Hide not recovered state (#36224)
This commit hides ClusterStates that have a STATE_NOT_RECOVERED_BLOCK from
ClusterStateAppliers. This is needed, because some appliers, such as IngestService, rely on
the fact, that cluster states with STATE_NOT_RECOVERED_BLOCK won't contain anything useful.
Once the state is recovered it's fully available for the appliers. This commit also switches many of
the remaining tests that require state persistence/recovery from Zen1 to Zen2.
2018-12-05 23:11:20 +01:00
jaymode b8743f6636
Test: mute testSnapshotAndRestoreWithNested
This mutes SourceOnlySnapshotIT#testSnapshotAndRestoreWithNested as it
fails reproducibly.

See #36276
2018-12-05 13:44:11 -07:00
Jim Ferenczi 18866c4c0b
Make hits.total an object in the search response (#35849)
This commit changes the format of the `hits.total` in the search response to be an object with
a `value` and a `relation`. The `value` indicates the number of hits that match the query and the
`relation` indicates whether the number is accurate (in which case the relation is equals to `eq`)
or a lower bound of the total (in which case it is equals to `gte`).
This change also adds a parameter called `rest_total_hits_as_int` that can be used in the
search APIs to opt out from this change (retrieve the total hits as a number in the rest response).
Note that currently all search responses are accurate (`track_total_hits: true`) or they don't contain
`hits.total` (`track_total_hits: true`). We'll add a way to get a lower bound of the total hits in a
follow up (to allow numbers to be passed to `track_total_hits`).

Relates #33028
2018-12-05 19:49:06 +01:00
Yannick Welsch cc11953724 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 16:55:45 +01:00
Tim Brooks 068c856e88
Rename internal repository actions to be internal (#36244)
This is a follow-up to #36086. It renames the internal repository
actions to be prefixed by "internal". This allows the system user to
execute the actions.

Additionally, this PR stops casting Client to NodeClient. The client we
have is a NodeClient so executing the actions will be local.
2018-12-05 08:11:47 -07:00
David Roberts a3c1c6938a Mute ForecastIT.testSingleSeries
Due to https://github.com/elastic/elasticsearch/issues/36258
2018-12-05 14:41:07 +00:00
Yannick Welsch b20497560c Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 14:06:38 +01:00
Adrien Grand 46209591fe
Propagate a doc-value `format` with inner hits. (#36068)
While sql passes a doc-value `format` when asking for doc-value fields, it
doesn't do it when asking for fields via inner hits.
2018-12-05 14:06:32 +01:00
Gil Raphaelli 37127c19ba
add missing error type mapping for apm-server monitoring (#36178) 2018-12-05 07:46:17 -05:00
Alan Woodward 73ceaad03a
Update to lucene-8.0.0-snapshot-c78429a554 (#36212)
Includes:

* A fix for a bug in Intervals.or() (https://issues.apache.org/jira/browse/LUCENE-8586)
* The ability to disable offset mangling in WordDelimiterGraphFilter
        (https://issues.apache.org/jira/browse/LUCENE-8509)
* BM25Similarity no longer multiplies scores by k1 + 1
2018-12-05 12:43:56 +00:00
Alan Woodward d2886e1c81
HLRC: execute watch API (#35868)
This change adds support for the execute watch API in the high level rest client
2018-12-05 12:41:42 +00:00
Martijn van Groningen a264cb6ddb
Refactor AutoFollowCoordinator to track leader indices per remote cluster (#36031)
and replaced poll interval setting with a hardcoded poll interval.
The hard coded interval will be removed in a follow up change to make
use of cluster state API's wait_for_metatdata_version.

Before the auto following was bootstrapped from thread pool scheduler,
but now auto followers for new remote clusters are bootstrapped when
a new cluster state is published.

Originates from #35895
Relates to #33007
2018-12-05 13:39:14 +01:00
Alpar Torok 60e45cd81d
Testing conventions task part 2 (#36107)
Closes #35435

- make it easier to add additional testing tasks with the proper configuration and add some where they were missing.
- mute or fix failing tests
- add a check as part of testing conventions to find classes not included in any testing task.
2018-12-05 14:20:01 +02:00
Martijn van Groningen 11935cd480
Replace Streamable w/ Writeable in BaseTasksResponse and subclasses (#36176)
This commit replaces usages of Streamable with Writeable for the
BaseTasksResponse / TransportTasksAction classes and subclasses of
these classes.

Note that where possible response fields were made final.

Relates to #34389
2018-12-05 13:14:10 +01:00
Yannick Welsch 42457b5960 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 11:39:38 +01:00
Martijn van Groningen b9707c29a1
[CCR] Change get autofollow patterns API response format (#36203)
The current response format is:

```
{
    "pattern1": {
        ...
    },
    "pattern2": {
        ...
    }
}
```

The new format is:

```
{
    "patterns": [
        {
            "name": "pattern1",
            "pattern": {
                ...
            }
        },
        {
            "name": "pattern2",
            "pattern": {
                ...
            }
        }
    ]
}
```

This format is more structured and more friendly for parsing and generating specs.
This is a breaking change, but it is better to do this now while ccr
is still a beta feature than later.

Follow up from #36049
2018-12-05 08:41:27 +01:00
Yogesh Gaikwad 090d766f35
[Kerberos] Find if port is available before using it for Kdc server (#36192)
If the randomly selected port was already in use the Kerberos
tests would fail. This commit adds check to see if the network
port is available and if not continue to find one for KDC server.
If it does not find port after 100 retries it throws an exception.

Closes #34261
2018-12-05 16:44:49 +11:00
Tim Vernum 3272538701
Make credentials mandatory when launching xpack/migrate (#36197)
Made credentials mandatory for xpack migrate tool. 
Closes #29847.

The x-pack user and roles APIs aren't available unless security is enabled, so the tool should always be called with the -u and -p options specified.
2018-12-05 15:51:50 +11:00
Tim Brooks 8bde608979
Register CcrRepository based on settings update (#36086)
This commit adds an empty CcrRepository snapshot/restore repository.
When a new cluster is registered in the remote cluster settings, a new
CcrRepository is registered for that cluster.

This is implemented using a new concept of "internal repositories".
RepositoryPlugin now allows implementations to return factories for
"internal repositories". The "internal repositories" are different from
normal repositories in that they cannot be registered through the
external repository api. Additionally, "internal repositories" are local
to a node and are not stored in the cluster state.

The repository will be unregistered if the remote cluster is removed.
2018-12-04 14:36:50 -07:00
Lisa Cawley e1fb1505f9
[DOCS] Moves security config file info (#36232) 2018-12-04 13:18:54 -08:00
Yannick Welsch 70c361ea5a Merge remote-tracking branch 'elastic/master' into zen2 2018-12-04 21:26:11 +01:00
Benjamin Trent c007a42ba0
[ML] adjusting post backport (#36222) 2018-12-04 13:29:29 -06:00
Adrien Grand d24b40f688
Make typeless APIs usable with indices whose type name is different from `_doc` (#35790)
This commit makes `document`, `update`, `explain`, `termvectors` and `mapping`
typeless APIs work on indices that have a type whose name is not `_doc`.
Unfortunately, this needs to be a bit of a hack since I didn't want calls with
random type names to see documents with the type name that the user had chosen
upon type creation.

The `explain` and `termvectors` do not support being called without a type for
now so the test is just using `_doc` as a type for now, we will need to fix
tests later but this shouldn't require further changes server-side since passing
`_doc` as a type name is what typeless APIs do internally anyway.

Relates #35190
2018-12-04 19:22:17 +01:00
Adrien Grand 0df08dd458
Set Lucene version upon index creation. (#36038)
It is important that all shards of a given index have the same
`indexCreatedVersionMajor` to Lucene, or eg. merging those shards is going to
be considered illegal. At the moment, we use the latest Lucene version when
creating a shard, which could cause shards to have different created versions
eg. in case of forced allocation. This commit makes sure to reuse the
appropriate Lucene version in order to avoid such issues.

Closes #33826
2018-12-04 17:53:20 +01:00
Benjamin Trent 166d9a94d4
[ML] Add lazy parsing for DatafeedConfig:Aggs,Query (#36117)
* Lazily parsing aggs and query in DatafeedConfigs

* Adding parser tests

* Fixing exception types && unneccessary checked ex

* Adding semi aggregation parser

* Adding tests, fixing up semi-parser

* Reverting semi-parsing

* Moving agg validations

* Making bad configs throw badRequestException
2018-12-04 09:41:47 -06:00
Martijn van Groningen 6e1ff31222
[CCR] AutoFollowCoordinator should tolerate that auto follow patterns may be removed (#35945)
AutoFollowCoordinator should take into account that after auto following
an index and while updating that a leader index has been followed, that
the auto follow pattern may have been removed via delete auto follow patterns
api.

Also fixed a bug that when a remote cluster connection has been removed,
the auto follow coordinator does not die when it tries get a remote client for 
that cluster.

Closes #35480
2018-12-04 15:55:15 +01:00
Yannick Welsch 80ee7943c9 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-04 09:37:09 +01:00
Andy Bristol a6ff9f7f66
[test] generate unique user names (#36179) 2018-12-03 15:08:00 -08:00
Tal Levy fdec331b13
[ILM] fix ilm.remove_policy rest-spec (#36165)
The rest interface for remove-policy-from-index API does not support
`_ilm/remove`, it requires that an `{index}` pattern be defined in
the URL path. This fixes the rest-api-spec to reflect the implementation
2018-12-03 10:55:31 -08:00
Julie Tibshirani 59ee8b5c69
Remove the deprecated _termvector endpoint. (#36131) 2018-12-03 10:22:42 -08:00
Jake Landis f8f521bad4
Deprecate /_xpack/monitoring/* in favor of /_monitoring/* (#36130)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.

* Add deprecation for /_xpack/monitoring/_bulk in favor of /_monitoring/bulk
* Removed xpack from the rest-api-spec and tests
* Removed xpack from the Action name
* Removed MonitoringRestHandler as an unnecessary abstraction
* Minor corrections to comments

Relates #35958
2018-12-03 10:26:08 -06:00
Jim Ferenczi 74aca756b8
Remove the distinction between query and filter context in QueryBuilders (#35354)
When building a query Lucene distinguishes two cases, queries that require to produce a score and queries that only need to match. We cloned this mechanism in the QueryBuilders in order to be able to produce different queries based on whether they need to produce a score or not. However the only case in es that require this distinction is the BoolQueryBuilder that sets a different minimum_should_match when a `bool` query is built in a filter context..
This behavior doesn't seem right because it makes the matching of `should` clauses different when the score is not required.

Closes #35293
2018-12-03 11:49:11 +01:00
Alpar Torok fa4d5f844d
Fix test fixtures on aufs (#36105)
Closes #36073

The problem showed up on debian 8 which uses aufs docker storage
driver by default as opposed to overlay2 used on other distros.
aufs does not support acls and thus the failure.
The --use-ntvfs option instructs samba not to rely on acls.
From what I can tell this is an implementation detail that should not
affect the tests ( which continue to pass )
2018-12-03 11:01:05 +02:00
Martijn van Groningen 43773a32a4
Replace Streamable w/ Writeable in BaseTasksRequest and subclasses (#35854)
* Replace Streamable w/ Writeable in BaseTasksRequest and subclasses

This commit replaces usages of Streamable with Writeable for the
BaseTasksRequest / TransportTasksAction classes and subclasses of
these classes.

Relates to #34389
2018-12-03 08:04:29 +01:00
Tim Vernum d20bb3789d
Add DEBUG/TRACE logs for LDAP bind (#36028)
Introduces a debug log message when a bind fails and a trace message
when a bind succeeds.

It may seem strange to only debug a bind failure, but failures of this
nature are relatively common in some realm configurations (e.g. LDAP
realm with multiple user templates, or additional realms configured
after an LDAP realm).
2018-12-03 10:05:57 +11:00
Lisa Cawley 46962308aa
[DOCS] Replace deprecated ldap setting (#36022) 2018-11-30 16:58:19 -08:00
Julie Tibshirani 0e1ddfd825
Deprecate types in document delete requests. (#36087)
* Make sure to use _doc as a type name in the CRUD HLRC tests.
* Deprecate types in document delete requests.
2018-11-30 15:11:29 -08:00
Tim Brooks ea7ea51050
Make `TcpTransport#openConnection` fully async (#36095)
This is a follow-up to #35144. That commit made the underlying
connection opening process in TcpTransport asynchronous. However the
method still blocked on the process being complete before returning.
This commit moves the blocking to the ConnectionManager level. This is
another step towards the top-level TransportService api being async.
2018-11-30 11:30:42 -07:00
Jim Ferenczi e179fd1274
Add support for rest_total_hits_as_int in watcher (#36035)
This change adds the support for rest_total_hits_as_int
in the watcher search inputs. Setting this parameter in the request
will transform the search response to contain the total hits as
a number (instead of an object).
Note that this parameter is currently a noop since #35849 is not
merged.

Closes #36008
2018-11-30 18:02:37 +01:00
Tim Brooks 370472b6d1
Upgrade Netty 4.3.32.Final (#36102)
This commit upgrades netty. This will close #35360. Netty started
throwing an IllegalArgumentException if a CompositeByteBuf is
created with < 2 components. Netty4Utils was updated to reflect this
change.
2018-11-30 09:02:10 -07:00
Jim Ferenczi 11fa5c626b [TEST] Fix random test failure in GetWatchResponseTests 2018-11-30 16:12:46 +01:00
Martijn van Groningen 32f7fbd9f0
[TEST] Set 'index.unassigned.node_left.delayed_timeout' to 0 in ccr tests
Some tests kill nodes and otherwise it would take 60s by default
for replicas to get allocated and that is longer than we wait
for getting in a green state in tests.

Relates to #35403
2018-11-30 11:03:36 +01:00
Jim Ferenczi 5c7b2c5f9b
HLRC: Add get watch API (#35531)
This changes adds the support for the get watch API in the high level rest client.
2018-11-30 11:02:46 +01:00
Dimitris Athanasiou 54cf1f9d74
[ML] Refactor control message writer to allow reuse for other processes (#36070) 2018-11-30 09:25:35 +00:00
Tal Levy 78e07d467c
[ILM] rest-spec for remove-policy had wrong link (#36083) 2018-11-29 14:11:51 -08:00
Albert Zaharovits 04ee05f199
Fix Watcher NotificationService's secure settings (#35610)
The NotificationService (base class for SlackService, HipchatService ...) has both dynamic
cluster settings and SecureSettings and builds the clients (Account) that are used to comm
with external services. This commit fixes an important bug about updating/reloading any
of these settings (both Secure and dynamic cluster). Briefly the bug is due to the fact that
both the secure settings as well as the dynamic node scoped ones can be updated
independently, but when constructing the clients some of the settings might not be visible.
2018-11-29 21:34:16 +02:00
Tal Levy fa5f551b80
[ILM] make alias swapping atomic (#35972)
In the current implementation, there is a time between the
ShrinkStep and the ShrinkSetAliasStep that both the source and
target indices will be present with the same aliases. This means
that queries to during this time will query both and return
duplicates. This fixes that scenario by moving the alias inheritance
to the same aliases update request as is done in ShrinkSetAliasStep
2018-11-29 11:11:20 -08:00
Jay Modi ba3ee98943
Security: improve exact index matching performance (#36017)
This commit improves the efficiency of exact index name matching by
separating exact matches from those that include wildcards or regular
expressions. Internally, exact matching is done using a HashSet instead
of adding the exact matches to the automata. For the wildcard and
regular expression matches, the underlying implementation has not
changed.
2018-11-29 11:37:06 -07:00
Zachary Tong 61c2db5ebb Revert "Deprecate X-Pack centric rollup endpoints (#35962)"
This reverts commit b84f1f6a3a.
2018-11-29 12:58:23 -05:00
Ryan Ernst 95d9cefea7
Build: Fix jdbc jar pom to not include deps (#36036)
This commit adds back the exclusion of dependencies from the pom file
for the jdbc jar, which was accidentally lost in #32014
2018-11-29 09:39:17 -08:00
Jim Ferenczi 8a7f3f75f3
Add support for rest_total_hits_as_int (#36051)
The support for rest_total_hits_as_int has already been merged to 6x
in #35848 so this change adds this new option to master. The plan was
to add this new option as part of #35848 but we've decided to wait a few
days before merging this breaking change so this commit just handles
the new option as a noop exactly like 6x for now. This will allow
users to migrate to this parameter before #35848 is merged.

Relates #33028
2018-11-29 18:36:16 +01:00
Costin Leau 1d458e3f60
SQL: Make INTERVAL millis optional (#36043)
Fractions of the second are not mandatory anymore inside INTERVAL
 declarations

Fix #36032
2018-11-29 18:52:47 +02:00
Tim Brooks c305f9dc03
Make keepalive pings bidirectional and optimizable (#35441)
This is related to #34405 and a follow-up to #34753. It makes a number
of changes to our current keepalive pings.

The ping interval configuration is moved to the ConnectionProfile.

The server channel now responds to pings. This makes the keepalive
pings bidirectional.

On the client-side, the pings can now be optimized away. What this
means is that if the channel has received a message or sent a message
since the last pinging round, the ping is not sent for this round.
2018-11-29 08:55:53 -07:00
Albert Zaharovits 5eb704040f
While most peoples' opinions change, the conviction of their correctness never does. (#35988) 2018-11-29 16:12:25 +02:00
Alpar Torok 9709282735
Don't disable tasks based on the plugin (#36045)
Some times the test fixtures plugin did not correctly disable tasks
from the build plugin as it should.
The plugin manager and tasks both use domain name collections so
the previus conde should have worked.
I did not have trime to track it down, but suspect there's some race
condition in Gradle causing this. The plugin manager is still incubating.
Since the tasks are on the cp even if the plugin is not applyed, we
don't really  need to involve the plugin at all.

Closes #36041
2018-11-29 15:54:34 +02:00
David Turner 7f257187af
[Zen2] Update default for USE_ZEN2 to true (#35998)
Today the default for USE_ZEN2 is false and it is overridden in many places. By
defaulting it to true we can be sure that the only places in which Zen2 does
not work are those in which it is explicitly set to false.
2018-11-29 12:18:35 +00:00
Jason Tedor 9fa9e1419f
Remove X-Pack centric graph endpoints (#36010)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-29 07:09:37 -05:00
Andrei Stefan 525c3b044a
Leftover from https://github.com/elastic/elasticsearch/pull/35964 (#36030) 2018-11-29 14:06:48 +02:00
Ioannis Kakavas 89c515175b Update version for authenticate API responses
This commits changes the serialization version from V_7_0_0 to
v_6_6_0 for the authenticate API response now that the work to add
the realm info in the response has been backported to 6.x in
b515ec7c9b9074dfa2f5fd28bac68fd8a482209e

Relates #35648
2018-11-29 13:31:52 +02:00
Alpar Torok c63d0af913 Prevent random build failures
Looks like some odd race condition causes failed builds by attempting to
run the task that should be disabled.
Disable the task explicitly untill we figure it out.
2018-11-29 11:30:09 +02:00
Martijn van Groningen 1390f366d4
[CCR] Only auto follow indices when all primary shards have started (#35814)
This change adds an extra check that verifies that all primary shards
have been started of an index that is about to be auto followed.

If not all primary shards have been started for an index
then the next auto follow run will try to follow to auto follow
this index again.

Closes #35480
2018-11-29 09:46:09 +01:00
Alpar Torok 45db829039
Replace fixtures with docker-compose (#35651)
Creates a new plugin to manage docker-compose based test fixtures. 
Convert the smb-fixture as a first example.
2018-11-29 09:43:16 +02:00
Tim Vernum 609f742e5f
Fix kerberos setting registration (#35986)
In #30241 Realm settings were changed, but the Kerberos realm settings
were not registered correctly. This change fixes the registration of
those Kerberos settings.

Also adds a new integration test that ensures every internal realm can
be configured in a test cluster.

Also fixes the QA test for kerberos.

Resolves: #35942
2018-11-29 18:06:52 +11:00
Alpar Torok e0a678f0c4
Remove version.qualified from MainResponse (#35412)
The fully qualified version will be returned as `version.number`
2018-11-29 08:41:39 +02:00
Gordon Brown c26af3b0a2
Deprecate X-Pack centric Migration endpoints (#35976)
This commit is part of our plan to deprecate and remove the use of
_xpack in the REST API routes.
2018-11-28 13:19:33 -07:00
Benjamin Trent 513e1ed095
ML: Adding XContentObjectTransformer class (#35957)
* ML: Adding XContentObjectTransformer class

* adding license headers

* Adding custom deprecation handler, and test for checking parsing failures

* forwarding deprection logs to LoggingDeprecationHandler
2018-11-28 11:43:57 -06:00
Nik Everett 0588dad80b
Tasks: Only require task permissions (#35667)
Right now using the `GET /_tasks/<taskid>` API and causing a task to opt
in to saving its result after being completed requires permissions on
the `.tasks` index. When we built this we thought that that was fine,
but we've since moved towards not leaking details like "persisting task
results after the task is completed is done by saving them into an index
named `.tasks`." A more modern way of doing this would be to save the
tasks into the index "under the hood" and to have APIs to manage the
saved tasks. This is the first step down that road: it drops the
requirement to have permissions to interact with the `.tasks` index when
fetching task statuses and when persisting statuses beyond the lifetime
of the task.

In particular, this moves the concept of the "origin" of an action into
a more prominent place in the Elasticsearch server. The origin of an
action is ignored by the server, but the security plugin uses the origin
to make requests on behalf of a user in such a way that the user need
not have permissions to perform these actions. It *can* be made to be
fairly precise. More specifically, we can create an internal user just
for the tasks API that just has permission to interact with the `.tasks`
index. This change doesn't do that, instead, it uses the ubiquitus
"xpack" user which has most permissions because it is simpler. Adding
the tasks user is something I'd like to get to in a follow up change.

Instead, the majority of this change is about moving the "origin"
concept from the security portion of x-pack into the server. This should
allow any code to use the origin. To keep the change managable I've also
opted to deprecate rather than remove the "origin" helpers in the
security code. Removing them is almost entirely mechanical and I'd like
to that in a follow up as well.

Relates to #35573
2018-11-28 09:28:27 -05:00
Marios Trivyzas a2338bb116
SQL: Fix issue with wrong data type for scripted Grouping keys (#35969)
When the grouping key of a GROUP BY is a painless script (functions are
involved), the data type of the key was incorrect in certain cases
(Boolean, IP, Date). This resulted in returning wrong data type for this
columns in the query results. E.g.:

```
SELECT COUNT(*), a > 10 AS a FROM t GROUP BY a
```

Fixes: #35662
2018-11-28 14:29:10 +01:00
Jason Tedor a3186e4a32
Deprecate X-Pack centric license endpoints (#35959)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-28 08:24:35 -05:00
Jason Tedor 78ac12d952
Remove deprecated Graph endpoints (#35956)
We had some endpoints in Graph that are deprecated for removal in
7.0.0. This commit removes these deprecated endpoints.
2018-11-28 08:24:15 -05:00
Andrei Stefan aabff7318e
SQL: DATABASE() and USER() system functions (#35946) 2018-11-28 15:20:04 +02:00
Vladimir Dolzhenko f0c5f0c099 [HLRC] XPack ML info action (#35777)
Relates to #29827
2018-11-28 10:58:20 +00:00
Costin Leau 95346926ef
SQL: Lock down JDBC driver (#35798)
Move classes under the same package to avoid internal classes being
 exposed to the outside. Remove public visibility outside 3 classes:
 EsDriver, EsDataSource and EsTypes.
The driver only has one package, namely org.elasticsearch.xpack.sql.jdbc
Use Es prefix for classes to ease name conflict and indicate their
 destination

Fix #35437
2018-11-28 12:48:26 +02:00
Christoph Büscher 2f547bac65
Remove deprecated methods from QueryStringQueryBuilder (#35912)
This change removes the deprecated useDisMax() and useAllFields() methods from
the QueryStringQueryBuilder and related tests. The disMax parameter has already
been a no-op since 6.0 and also the useAllFields has been deprecated since 6.0
and there is a direct replacement via defaultField.
2018-11-28 11:09:03 +01:00
Ioannis Kakavas d948edfd69 [TESTS] Mute SmokeTestWatcherWithSecurityIT tests
Issue tracked in
https://github.com/elastic/elasticsearch/issues/35361
2018-11-28 11:34:41 +02:00
Ioannis Kakavas fc7e7e7d81 [TESTS] Mute SNI tests on FIPS 140 JVMs
These tests use a TrustAllConfig TrustManager as they set
verification_mode to none, that can't be used in a FIPS 140 JVM.
2018-11-28 09:35:36 +02:00
Jason Tedor c42d9d91c9
Deprecate X-Pack centric SQL endpoints (#35964)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-27 22:16:21 -05:00
Yogesh Gaikwad e50e0f997a
[Kerberos] Add support for Kerberos V5 Oid (#35764)
Clients can use the Kerberos V5 security mechanism and when it
used this to establish security context it failed to do so as
Elasticsearch server only accepted Spengo mechanism.
This commit adds support to accept Kerberos V5 credentials
over spnego.

Closes #34763
2018-11-28 13:29:43 +11:00
Jason Tedor b84f1f6a3a
Deprecate X-Pack centric rollup endpoints (#35962)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-27 20:34:17 -05:00
Albert Zaharovits 36819f78ef
DOCS Audit event attributes in new format (#35510)
Accounts for the `Structured Audit Entries` in the format
documentation.
2018-11-28 01:24:03 +02:00
Ioannis Kakavas 580b5baf21
Add realm information for Authenticate API (#35648)
- Add the authentication realm and lookup realm name and type in the response for the _authenticate API
- The authentication realm is set as the lookup realm too (instead of setting the lookup realm to null or empty ) when no lookup realm is used.
2018-11-27 23:35:42 +02:00
Zachary Tong 1a1092d08f [Rollup] Update serialization version after backport 2018-11-27 16:02:48 -05:00
Zachary Tong 48fa251812
[Rollup] Add more diagnostic stats to job (#35471)
* [Rollup] Add more diagnostic stats to job

To help debug future performance issues, this adds the
 min/max/avg/count/total latencies (in milliseconds) for search
and bulk phase.  This latency is the total service time including
transfer between nodes, not just the `took` time.

It also adds the count of search/bulk failures encountered during
runtime.  This information is also in the log, but a runtime counter
will help expose problems faster

* review cleanup

* Remove dead ParseFields
2018-11-27 15:46:10 -05:00
Chris Earle 3337fa7351
[Monitoring] Make Exporters Async (#35765)
This changes the exporter code -- most notably the `http` exporter --
to use async operations throughout the resource management and bulk
initialization code (the bulk indexing of monitoring documents was
already async).

As part of this change, this does change one semi-core aspect of the
`HttpResource` class in that it will no longer block all concurrent calls
until the first call completes with
`HttpResource::checkAndPublishIfDirty`.
Now, any parallel attempts to check the resources will be skipped until
the first call completes (success or failure). While this is a technical
change, it has very little practical impact because the existing behavior
was either quick success (then every blocked request processed) or
each request timed out and failed anyway, thus being effectively
skipped (and a burden on the system).
2018-11-27 14:28:14 -05:00
Tal Levy c2329cdf1d
[ILM] reduce time restriction on IndexLifecycleExplainResponse (#35954)
step times were set. The assumption was that these are always set.
Tests passed, which led me to believe this was true. There is a time
when shrunk indices have their step phase/action/step details set,
but with no time information (in the CopyExecutionStateStep).
Explain API fails for these
2018-11-27 11:03:47 -08:00
Jay Modi 2061eeb122
Remove use of AbstractComponent in xpack (#35394)
This commit removes the use of AbstractComponent in xpack where it was
still being extended. It has been replaced with explicit logger
declarations.

See #34488
2018-11-27 11:28:26 -07:00
Luca Cavanna 7e772e311d
Remove fromXContent from IndexUpgradeInfoResponse (#35934)
Such method is there only for testing purposes, it is not needed.
2018-11-27 19:04:44 +01:00
Tim Brooks b6ed6ef189
Add sni name to SSLEngine in nio transport (#35920)
This commit is related to #32517. It allows an "sni_server_name"
attribute on a DiscoveryNode to be propagated to the server using
the TLS SNI extentsion. Prior to this commit, this functionality
was only support for the netty transport. This commit adds this
functionality to the security nio transport.
2018-11-27 09:06:52 -07:00
Tal Levy df341aba4a
[ILM] make IndexLifecycleExplainResponse more resilient to null values (#35800)
added validation for complete information of step details.

also changed the rendering of explain responses so null strings are not rendered

Another thing that I changed is the format of the client-side response. I found it difficult to maintain the two subtly-different objects, so I migrated the usage of long for the fields, to Long (just as it is on the server-side).
2018-11-27 07:29:59 -08:00
Alexander Reelsen a615ab454a
Watcher: Only trigger a watch if new or schedule/changed (#35908)
The trigger engine did always create a new schedule data structure, when
the watcher indexing listener called an add. However the indexing
listener also called add, when the watch status was updated. This means,
that upon a watch status update the watch got retriggered, potentially
waiting a defined interval from the watch status update onwards, instead
of waiting from the last run.

This commit only updates the schedule in the trigger engine, if it
actually has changed, otherwise the existing schedule will not be
touched. This has two results

1. If a watch is updated by an execution, the existing interval will not
be touched (meaning the scheduled time will not move forward).
2. If a watch is updated by a user, but the schedule is not changed, it
will not be reset from the update (for example starting to count from 5
minutes again, if the interval was set to 5 minutes).

Furthermore some minor cleanups were applied, making variables final in
the ctor, preventing double creation of variables.
2018-11-27 16:15:37 +01:00
Marios Trivyzas 1da9c6faa0
SQL: Fix translation of math functions to painless (#35910)
`SIGN` and `RADIANS` where wrongly overriding `mathFunction()`.
Converted `mathFunction()` to private in `MathFunction` since it
shouldn't be overriden, as it uses the assigned `MathOperation`
to get the funtion name for painless scripts.

Fixes: #35654
2018-11-27 15:33:43 +01:00
Marios Trivyzas c91ef1105d
SQL: Implement data type verification for conditionals (#35916)
Add special verifier rule to check that the arguments of conditional
functions are of the same or compatible types. This way the user gets
a descriptive error message with line number and column indicating
where is the offending argument.

Closes: #35907
2018-11-27 14:46:11 +01:00
David Roberts 110c4fdd65
[ML] Adjust file structure finder parser config (#35935) 2018-11-27 12:52:52 +00:00
Tim Vernum b6eb73a835
Update BWC checks in HasPrivilegesResponse (#35929)
This serialization has been backported to 6.x
2018-11-27 21:05:12 +11:00
Ioannis Kakavas 998c663f0d
Add Tests for findSamlRealm (#35905)
This commit adds a test for handling correctly all they possible 
`SamlPrepareAuthenticationRequest` parameter combinations that 
we might get from Kibana or a custom web application talking to the
SAML APIs. 
We can match the correct SAML realm based either on the realm name
or the ACS URL. If both are included in the request then both need to 
match the realm configuration.
2018-11-27 08:18:18 +02:00
Tim Vernum 5b427d415e
Add "request.id" to file audit logs (#35536)
This generates a synthesized "id" for each incoming request that is
included in the audit logs (file only).
This id can be used to correlate events for the same request (e.g.
authentication success with access granted).

This request.id is specific to the audit logs and is not used for any
other purpose

The request.id is consistent across nodes if a single request requires
execution on multiple nodes (e.g. search acros multiple shards).
2018-11-27 15:19:47 +11:00
Tim Vernum a18b219f79
Allow noop PutUser updates (#35843)
When assertions are enabled, a Put User action that have no effect (a
noop update) would trigger an assertion failure and shutdown the node.

This change accepts "noop" as an update result, and adds more
diagnostics to the assertion failure message.
2018-11-27 15:08:53 +11:00
Ryan Ernst 10447dd962
Build: Fix jdbc jar to include deps (#35602)
This commit adds back bundling of all deps of the sql jdbc jar. This was
lost in a refactoring of how the shadow plugin is handled for the entire
elasticsearch project.
2018-11-26 16:18:35 -08:00
Gordon Brown 119835decd
Always enforce cluster-wide shard limit (#34892)
This removes the option to run a cluster without enforcing the
cluster-wide shard limit, making strict enforcement the default and only
behavior.  The limit can still be adjusted as desired using the cluster
settings API.
2018-11-26 17:05:12 -07:00
Tal Levy 8fe5f66637
[ILM] fix rest-api-spec doc links (#35478) 2018-11-26 12:26:36 -08:00
Marios Trivyzas 3f7cae3f0d
SQL: Implement GREATEST and LEAST functions (#35879)
Add GREATEST(expr1, expr2, ... exprN) and LEAST(expr1, expr2, exprN)
functions which are in the family of CONDITIONAL functions.

Implementation follows PostgreSQL behaviour, so the functions return
`NULL` when all of their arguments evaluate to `NULL`.

Renamed `CoalescePipe` and `CoalesceProcessor` to `ConditionalPipe` and
`ConditionalProcessor` respectively, to be able to reuse them for
`Greatest` and `Least` evaluations. To achieve that `ConditionalOperation`
has been added to differentiate between the functionalities at execution
time.

Closes: #35878
2018-11-26 18:21:36 +01:00
Costin Leau d291b080de
SQL: SYS COLUMNS returns ODBC specific schema (#35870)
Due to some unresolvable type conflict between the expected definition
 in JDBC vs ODBC, the driver mode is now passed over so that certain
 command can change their results accordingly (in this case SYS COLUMNS)

Fix #35376
2018-11-26 16:59:09 +02:00
Marios Trivyzas b078e2970c
SQL: Implement null safe equality operator `<=>` (#35873)
This operator handles nulls in different way than the normal `=`.
If one of the operants is `null` and the other not it returns `false`.
If both operants are `null` it returns `true`. Therefore in contrary to
`=`, which returns `null` if at least one of the operants is `null`, this one
never returns `null` as a result.

Closes: #35871
2018-11-26 14:02:02 +01:00
Costin Leau cd822b7ca8 SQL: Fix incorrect merging of #35852 2018-11-26 12:41:43 +02:00
Costin Leau 633a562208 SQL: Fix incorrect merging of #35852 2018-11-26 12:35:46 +02:00
Costin Leau 971299baf4
SQL: Add filtering to SYS TYPES (#35852)
Fix #35342
2018-11-26 11:26:24 +02:00
Andrei Stefan 00e6fec718
SQL: XPack FeatureSet functionality (#35725)
* Introduced "client.id" parameter for REST requests
* Bug that made the Verifier run twice, fixed in the Analyzer
* Single node IT and unit testing
2018-11-26 07:11:43 +02:00
Simon Willnauer 4711c5cdf3
Always return false from `refreshNeeded` on ReadOnlyEngine (#35837)
Acquiring a searcher is unnecessary to determine if a refresh is
necessary since read-only engines never refresh.

Closes #35785
2018-11-24 09:25:42 +01:00
Simon Willnauer e46e44ce38
Wrap can_match reader with ElasticsearchDirectoryReader (#35857)
Code that operates on-top of the engine requires all readers returned to be
unwrapped into ElasticsearchDirectoryReader. The special reader
the FrozenEngine uses wasn't wrapped.
2018-11-24 09:23:53 +01:00
Marios Trivyzas 410f570d5f
SQL: Implement NULLIF(expr1, expr2) function (#35826)
NULLIF returns null if the 2 expressions are equal or the
expr1 otherwise.

Closes: #35818
2018-11-23 22:19:27 +01:00
Costin Leau e22a83554e
SQL: Polish grammar for intervals (#35853) 2018-11-23 17:14:52 +02:00
Christoph Büscher 6fb7ce66e6
[Tests] Fix creating ExplainLifecycleRequest with no indices (#35828)
We didn't check that the ExplainLifecycleRequest was constructed with at least
one index before, now that we do we must also make sure the tests
mutateInstance() method used in equals/hashCode checks doesn't accidentally
create an empty index array.

Closes #35822
2018-11-23 05:03:19 +01:00
Jim Ferenczi e37a0ef844
Upgrade to lucene-8.0.0-snapshot-67cdd21996 (#35816) 2018-11-22 15:42:59 +01:00
Marios Trivyzas 92acf47c16
SQL: Implement NVL(expr1, expr2) (#35794)
Add NVL as alias to IFNULL as they have the same
behaviour. Add basic tests and docs.

Closes: #35782
2018-11-22 11:41:00 +01:00
Jason Tedor 2887680acb
Avoid NPE in follower stats when no tasks metadata (#35802)
When there is no persistent tasks metadata we could hit a null pointer
exception when executing a follower stats request. This is because we
inspect the persistent tasks metadata. Yet, if no tasks have been
registered, this is null (as opposed to empty). We need to avoid
de-referencing the persistent tasks metadata in this case. That is what
this commit does, and we add a test for this situation.
2018-11-21 19:16:28 -05:00
Zachary Tong b01daedfb5 [TEST] Ensure job does not trigger
By setting the cron to 2017, we ensure it won't trigger.  This makes it
easier to test because we know the job will _only_ be in STARTED,
and we can ignore INDEXING states due to transient triggers.

Closes #35779
2018-11-21 18:18:00 -05:00
Tal Levy 927b2425b6
[TEST] fix invalid ExplainLifecycleRequest in tests (#35804) 2018-11-21 14:59:08 -08:00
Marios Trivyzas d95d885bae
SQL: Implement ISNULL(expr1, expr2) (#35793)
Add ISNULL as an alias of IFNULL as they have the
same behaviour. Add basic test and docs.

Closes: #35781
2018-11-21 23:15:10 +01:00
Gordon Brown d9c6986b75
Ensure all Steps have Javadoc (#35757)
Adds or corrects Javadoc on subclasses of Step.
2018-11-21 10:06:57 -07:00
Marios Trivyzas e179bd393d
SQL: Implement IFNULL variant of COALESCE (#35762)
IFNULL is a MySQL variant (also used in other DBs) which
takes only 2 arguments and returns the first one that is not null.

Closes: #35749
2018-11-21 17:07:07 +01:00
Andrey Ershov a056bd8c1c
[Zen2] Move ClusterState fields to be persisted to ClusterState.MetaData (#35625)
Today we have a way to atomically persist global MetaData and
IndexMetaData to disk when new ClusterState is received. All other
ClusterState fields are not persisted.
However, there are other parts of ClusterState that should be
persisted, namely:

version
term
lastCommittedConfiguration
lastAcceptedConfiguration
votingTombstones

version is changed frequently, other fields are not. We decided
to group term, lastCommittedConfiguration,
lastAcceptedConfiguration and votingTombstones into
CoordinationMetaData class and make CoordinationMetaData a field
inside MetaData.
MetaData.toXContent and MetaData.fromXContent should take care of
CoordinationMetaData.
version stays as a top level field in ClusterState and will be
persisted as part of Manifest in a follow-up commit.
Also MetaData.isGlobalStateEquals should be extended to include
coordinationMetaData in comparison.

This commit favors exposing getters, such as getTerm directly in
ClusterState to avoid massive code changes.

An example of CoordinationMetaState.toXContent:

{
  "term": 1,
  "last_committed_config": [
    "TiIuBcbBtpuXyDDVHXeD",
    "ZIAoVbkjjLPLUuYLaTkw"
  ],
  "last_accepted_config": [
    "OwkXbXZNOZPJqccdFHdz",
    "LouzsGYwmQzpeQMrboZe",
    "fCKGRZdjLTqzXAqPUtGL",
    "pLoxshjpJXwDhbgjfYJy",
    "SjINLwFIlIEFZCbjrSFo",
    "MDkVncJEVyZLJktopWje"
  ]
}
2018-11-21 17:03:26 +01:00
Costin Leau 03f003733d
SQL: Perform lazy evaluation of mismatched mappings (#35676)
Move away from performing eager, fail-fast validation of mismatched
 mapping to a lazy evaluation based on the fields actually used in the
 query. This allows queries to run on the parts of the indices that
 "work" which is not just convenient but also a necessity for large
 mappings (like logging) where alignment is hard/impossible to achieve.

Fix #35659
2018-11-21 17:35:15 +02:00
Albert Zaharovits 592a909ba7
`manage_token` privilege for `kibana_system`
Creates the manage_token cluster privilege and adds it to the
kibana_system role. This is required if kibana were to use the token
service for its authenticator process.
Because kibana_system already has manage_saml this effectively
only adds the privilege to create tokens.
2018-11-21 17:15:14 +02:00
Andrey Ershov 6ac0cb1842 Merge branch master into zen2
2 types of conflicts during the merge:
1) Line length fix
2) Classes no longer extend AbstractComponent
2018-11-21 15:36:49 +01:00
Costin Leau f0a3d32602
SQL: Introduce INTERVAL support (#35521)
Introduce INTERVAL as a DataType
Add INTERVAL to the grammar which supports the standard SQL declaration
 (without precision):
 > INTERVAL '1 23:45:01.123456789' DAY TO SECOND
 but also number for single unit intervals:
 > INTERVAL 1 YEAR
 as well as the plurals of the units:
 > INTERVAL 2 YEARS
Interval are internally supported as just another Literal being backed
 by java.time.Period and java.time.Duration
Move JDBC away from JDBCType enum to SQLType interface
Refactor DataType by moving it into server core and adding dedicated (and
 much simpler) JDBC driver type
Improve internal JDBC conversion by normalizing on the DataType
Rename JDBC columnInfo to JdbcColumnInfo to differentiate between it and
 the SQL ColumnInfo

Fix #29990
2018-11-21 15:45:59 +02:00
Christoph Büscher 6638708b56
Remove deprecated QueryStringQueryBuilder#splitOnWhiteSpace (#35763)
This parameter has been deprecated and was ignored since 6.0, so its Java API
methods can be removed.
2018-11-21 10:29:08 +01:00
Luca Cavanna 65d8fdf3da
Clean up PutLicenseResponse (#35689)
This commit removes the parsing code from the PutLicenseResponse server variant, and the toXContent portion from the corresponding client variant.

Relates to #35547
2018-11-21 10:21:45 +01:00
Luca Cavanna 778550a97d
Clean up StartBasicResponse (#35688)
This commit removes the parsing code from the PostStartBasicResponse server variant. It also makes the server response implement StatusToXContent which allows us to save a couple of lines of code in the corredponding REST action.

Relates to #35547
2018-11-21 10:21:10 +01:00
Ioannis Kakavas e8ec4fad7b
[DOCS] Adjust Invalidate Token REST API docs (#35622)
- Renames API to Invalidate Token
- Explicitly calls out the possibility to invalidate refresh tokens
via this API
2018-11-21 09:32:56 +02:00
Tim Vernum 30c5422561
Move XContent generation to HasPrivilegesResponse (#35616)
The RestHasPrivilegesAction previously handled its own XContent
generation. This change moves that into HasPrivilegesResponse and
makes the response implement ToXContent.

This allows HasPrivilegesResponseTests to be used to test
compatibility between HLRC and X-Pack internals.

A serialization bug (cluster privs) was also fixed here.
2018-11-21 14:33:10 +11:00
Marios Trivyzas b1818dbdce
SQL: [docs] Add documentation for COALESCE (#35740)
Follows: #35253
2018-11-21 01:43:05 +01:00
Armin Braun 89cf4a7397
NETWORKING: Fix IpFiltering Test (#35730)
* The port assigned to all loopback interfaces doesn't necessarily have to be the same  for ipv4 and ipv6
=> use actual address from profile instead of just port + loopback in test
* Closes #35584
2018-11-20 17:50:08 +01:00
Christoph Büscher e91f404d16
Remove remains of 'auto_generate_phrase_queries' (#35735)
This parameter in the `query_string` query was deprecated in 6.0 and ignored
since then. Its API methods and remaining uses can be removed in the upcoming
major version.

Relates to #35734
2018-11-20 16:07:17 +01:00
Alpar Torok 8659af68e0
Auto skip license headers on no source (#35640)
* Unmute BuildExamplePluginsIT

* Skip licenseHeaders when there are no sources
2018-11-20 13:02:33 +02:00
Simon Willnauer 29ef442841
Add a `_freeze` / `_unfreeze` API (#35592)
This commit adds a rest endpoint for freezing and unfreezing an index.
Among other cleanups mainly fixing an issue accessing package private APIs
from a plugin that got caught by integration tests this change also adds
documentation for frozen indices.
Note: frozen indices are marked as `beta` and available as a basic feature.

Relates to #34352
2018-11-20 08:03:24 +01:00
Tim Brooks a989b675b5
Remove NPE from IndexFollowingIT (#35717)
Currently there is a common NPE in the IndexFollowingIT that does not
indicate the test failing. This is when a cluster state listener is
called and certain index metadata is not yet available.

This commit checks that the metadata is not null before performing the
logic that depends on the metadata.
2018-11-19 20:38:49 -07:00
Tim Vernum d367894c31
DOCS: Add password_hash & refresh to Put User API (#35574)
PR #35242 formalised support for the password_hash field in the body
of the Put User security API.
Since this field is now validated and tested, it can also be
documented.

The Put User API also supports a "refresh" query parameter that was
not documented. This commit adds it to the docs.
2018-11-20 13:03:22 +11:00
Tim Vernum 3bba9a56de
TEST: Enable skipped test (#35682)
This test was skipped due to infr complications during the work
to open x-pack, but wasn't enabled again when the issue was resolved.

Relates: #29890
2018-11-20 12:51:01 +11:00
Yannick Welsch 47ada69c46
Zen2: Move most integration tests to Zen2 (#35678)
Zen2 is now feature-complete enough to run most ESIntegTestCase tests. The changes in this PR
are as follows:
- ClusterSettingsIT is adapted to not be Zen1 specific anymore (it was using Zen1 settings).
- Some of the integration tests require persistent storage of the cluster state, which is not fully
implemented yet (see #33958). These tests keep running with Zen1 for now but will be switched
over as soon as that is fully implemented.
- Some very few integration tests are not running yet with Zen2 for other reasons, depending on
some of the other open points in #32006.
2018-11-19 21:15:29 +01:00
lcawl 3a232d7897 [DOCS] Adds anchor for SAML Kibana basic info 2018-11-19 11:42:45 -08:00
Benjamin Trent 80bcf7c704
[ML]: adjusting post backport of (#35645) (#35696) 2018-11-19 12:49:07 -06:00
Gordon Brown b2057138a7
Remove AbstractComponent from AbstractLifecycleComponent (#35560)
AbstractLifecycleComponent now no longer extends AbstractComponent. In
order to accomplish this, many, many classes now instantiate their own
logger.
2018-11-19 09:51:32 -07:00
Tanguy Leroux a8433a3164
[Docs] Fix `service_api_key` in PagerDuty documentation (#35589)
closes #35572
2018-11-19 16:30:20 +01:00
Benjamin Trent 6d4a3f8fce
Removes two unused AnalysisConfig options (#35645)
* ML: Removing result_finalization_window && overlapping_buckets

* Reverting bad method deletions

* Setting to current before backport to try and get a green build

* fixing testBuildAutodetectCommand test

* disabling bwc tests for backport
2018-11-19 08:29:53 -06:00
Benjamin Trent bc7dea4480
ML: changing automatic check_window calculation (#35643)
* ML: changing automatic check_window calculation

* adding docs on how we calculate the default
2018-11-19 08:03:34 -06:00
David Roberts f70b7dc158 Mute DataTypeConversionTests.testConversionToDate
Due to https://github.com/elastic/elasticsearch/issues/35683
2018-11-19 12:55:33 +00:00
David Roberts 2ee25c858d Mute CronEvalToolTests.testEnsureDateIsShownInRootLocale
Due to https://github.com/elastic/elasticsearch/issues/35687
2018-11-19 09:35:29 +00:00
Costin Leau 4119409b6d
SQL: Improve validation of unsupported fields (#35675)
Fix bug in Analyzer that caused it to report unsupported fields only
 when declared in projections. The rule has been extended to all field
 declarations.

Fix #35673
2018-11-18 23:35:18 +02:00
Costin Leau f8e333b117
SQL: Move internals from Joda to java.time (#35649)
Remove/Limit usage of Joda through-out the processors and functions
Use ZonedDateTime wherever possible instead of long/tzId

Fix #35633
2018-11-17 15:30:27 +02:00
Gordon Brown cce9648f9d
Align RolloverStep's name with other step names (#35655)
RolloverStep previously had a name of "attempt_rollover", which was
inconsistent with all other step names due it its use of an underscore
instead of a dash.
2018-11-16 17:42:48 -07:00
Ryan Ernst 3c03975ce3
Watcher: Use joda method to get local TZ (#35608)
This commit switches from using java util's default timezone method to
using joda. The former can cause problems when the string representation
of the timezone is unknown to joda.

closes #35518
2018-11-16 15:46:27 -08:00
Arthur Gavlyukovskiy 022726011c Remove use of AbstractComponent in server (#35444)
Removed extending of AbstractComponent and changed logger usage to
explicit declaration. Abstract classes still have logger
declaration using this.getClass() in order to show implementation class
name in its logs.

See #34488
2018-11-16 16:10:32 -05:00
Julie Tibshirani c6a0904e0e
Deprecate types in count and msearch. (#35421)
* Deprecate types in count requests.
* Move RestCountAction to the 'search' package.
* Deprecate types in multi search requests.
* Add tests for types deprecation in the _search endpoint.
2018-11-16 13:04:43 -08:00
Benjamin Trent e360c600d0
ML: Adjusing BWC version post backport to 6.6 (#35605) 2018-11-16 06:50:12 -06:00
Gordon Brown 3883e9bf4c
Split RolloverStep into Wait and Action steps (#35524)
RolloverAction will now periodically check the rollover conditions using
the Rollover API with the dry_run option as an AsyncWaitStep, then run
the rollover itself by calling the Rollover API with no conditions,
which will always roll over, as an AsyncActionStep. This will resolve
race condition issues in policies using RolloverAction.
2018-11-15 17:11:31 -07:00
David Turner 86ef041539
[Zen2] Introduce ClusterBootstrapService (#35488)
Today, the bootstrapping of a Zen2 cluster is driven externally, requiring
something else to wait for discovery to converge and then to inject the initial
configuration. This is hard to use in some situations, such as REST tests.

This change introduces the `ClusterBootstrapService` which brings the bootstrap
retry logic within each node and allows it to be controlled via an (unsafe)
node setting.
2018-11-15 20:09:22 +00:00
Benjamin Trent f7ada9b29b
Add delayed datacheck to the datafeed job runner (#35387)
* ML: Adding missing datacheck to datafeedjob

* Adding client side and docs

* Making adjustments to validations

* Making values default to on, having more sensible limits

* Intermittent commit, still need to figure out interval

* Adjusting delayed data check interval

* updating docs

* Making parameter Boolean, so it is nullable

* bumping bwc to 7 before backport

* changing to version current

* moving delayed data check config its own object

* Separation of duties for delayed data detection

* fixing checkstyles

* fixing checkstyles

* Adjusting default behavior so that null windows are allowed

* Mentioning the default value

* Fixing comments, syncing up validations
2018-11-15 13:32:45 -06:00
Lee Hinman 90b38d9b3d
[ILM] Avoid NullPointerException in CopyExecutionStateStep (#35568)
In the event that the target index does not exist when `CopyExecutionStateStep`
executes, this avoids a `NullPointerException` and provides a more helpful error
to the ILM user.

Resolves #35567
2018-11-15 09:29:26 -07:00
Tim Vernum 1e60b282fc Grant .tasks access to kibana_system role (#35573)
Kibana now uses the tasks API to manage automatic reindexing of the
.kibana index during upgrades.

The implementation of the tasks API requires that
1. the user executing the task can create & write to the ".tasks" index
2. the user checking on the status of the task can read (Get) the
   relevant document from the ".tasks" index
2018-11-15 08:14:58 -07:00
Andrei Stefan eaf010cd73
SQL: clear the cursor if nested inner hits are enough to fulfill the query required limits (#35398) 2018-11-15 14:33:47 +02:00
Luca Cavanna 8e2c84ad8b
Clean up XPackInfoResponse class and related tests (#35547)
Response classes in Elasticsearch (and xpack) only need to implement ToXContent, which is needed to print their output put in the REST layer and return the response in json (or others) format. On the other hand, response classes that are added to the high-level REST client, need to do the opposite: parse xcontent and create a new object based on that.

This commit removes the parsing code from the XPackInfoResponse server variant, and the toXContent portion from the corresponding client variant. It also removes a client specific test class that looks redundant now that we have a single test class for both classes.
2018-11-15 09:59:27 +01:00
Tanguy Leroux c9b4ef0dfd
Use RunOnce when appropriate (#35553)
This pull request replaces some blocks of code that must be run once 
and that are currently based on AtomicBoolean by the convient RunOnce 
class added in #35489.
2018-11-15 09:24:40 +01:00
David Turner 135c3f0f07 Merge branch 'master' into zen2 2018-11-15 08:24:26 +00:00
Jay Modi faa9523d19
Remove deprecated constructor from failure handler (#35565)
The DefaultAuthenticationFailureHandler has a deprecated constructor
that was present to prevent a breaking change to custom realm plugin
authors in 6.x. This commit removes the constructor and its uses.
2018-11-14 17:04:33 -07:00
Tal Levy 5d0544d732
update move-to-step spec with proper docs and extra warning to clients (#35477) 2018-11-14 14:08:31 -08:00
Lee Hinman 8ea999e489
Include stack trace with ILM error in explain output (#35512)
This changes the stacktrace to be included with the ILM explain error when the
index is an on ERROR step.

Before:

```json
{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date_millis" : 1542131670601,
      "phase" : "warm",
      "phase_time_millis" : 1542131676335,
      "action" : "shrink",
      "action_time_millis" : 1542131676335,
      "step" : "ERROR",
      "step_time_millis" : 1542131676451,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1542131669839
      }
    }
  }
}
```

After

```
{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date_millis" : 1542131670601,
      "phase" : "warm",
      "phase_time_millis" : 1542131676335,
      "action" : "shrink",
      "action_time_millis" : 1542131676335,
      "step" : "ERROR",
      "step_time_millis" : 1542131676451,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]",
        "stack_trace" : "java.lang.IllegalArgumentException: the number of target shards [13] must be less that the number of source shards [2]\n\tat org.elasticsearch.cluster.metadata.IndexMetaData.selectShrinkShards(IndexMetaData.java:1509)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction.prepareCreateIndexRequest(TransportResizeAction.java:146)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction$1.onResponse(TransportResizeAction.java:104)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction$1.onResponse(TransportResizeAction.java:101)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:60)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:383)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onNodeResponse(TransportBroadcastByNodeAction.java:352)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:324)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:314)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1117)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1198)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1178)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:417)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:391)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:714)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:726)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1542131669839
      }
    }
  }
}
```

Resolves #35498
2018-11-14 14:40:05 -07:00
Zachary Tong c7ea0343d2 [Rollup] Update versions after backport
Both serialization and yaml test skips
2018-11-14 09:56:45 -05:00
Andrey Ershov 045fdd0d3b Merge master into zen2 2018-11-14 15:37:13 +03:00
Tim Vernum 231f6c1595
Formal support for "password_hash" in Put User (#35242)
For some time, the PutUser REST API has supported storing a pre-hashed
password for a user. The change adds validation and tests around that
feature so that it can be documented & officially supported.

It also prevents the request from containing both a "password" and a "password_hash".
2018-11-14 16:46:31 +11:00
Zachary Tong c346a0f027
[Rollup] Add `wait_for_completion` option to StopRollupJob API (#34811)
This adds a `wait_for_completion` flag which allows the user to block 
the Stop API until the task has actually moved to a stopped state, 
instead of returning immediately.  If the flag is set, a `timeout` parameter
can be specified to determine how long (at max) to block the API
call.  If unspecified, the timeout is 30s.

If the timeout is exceeded before the job moves to STOPPED, a
timeout exception is thrown.  Note: this is just signifying that the API
call itself timed out.  The job will remain in STOPPING and evenutally
flip over to STOPPED in the background.

If the user asks the API to block, we move over the the generic
threadpool so that we don't hold up a networking thread.
2018-11-13 16:37:17 -05:00
David Turner 0e1a12122c Merge branch 'master' into zen2 2018-11-13 15:25:35 +00:00
Luca Cavanna 0b7d18d733
HLRC: add support for get license basic/trial status API (#33176)
Relates to #29827
2018-11-13 15:52:59 +01:00
Martijn van Groningen 0487181d0f
[TEST] Force flush to ensure multiple segments.
Relates to #35333
2018-11-13 14:58:17 +01:00
Simon Willnauer 3229dfc4de
Allow efficient can_match phases on frozen indices (#35431)
This change adds a special caching reader that caches all relevant
values for a range query to rewrite correctly in a can_match phase
without actually opening the underlying directory reader. This
allows frozen indices to be filtered with can_match and in-turn
searched with wildcards in a efficient way since it allows us to
exclude shards that won't match based on their date-ranges without
opening their directory readers.

Relates to #34352
Depends on #34357
2018-11-13 14:53:55 +01:00
David Roberts 09965cb370
[ML] Fix find_file_structure NPE with should_trim_fields (#35465)
The NPE would occur if should_trim_field was overridden to
true and any field value was completely blank.  This change
defends against this situation.

Fixes #35462
2018-11-13 08:49:24 +00:00
Tal Levy 16cbbab7b7
[ILM] fix retry so it picks up latest policy and executes async action (#35406)
Before, moving to a failed step would only change the step info
to be that of the failed step. This means two things.

1. Async Steps would never be triggered to execute
2. If there are inherent problems with the action definition that can
be fixed with a policy update, these changes were not being reflected
by the new execution info.

Changes now

1. Async steps are executed after the move to the failed step in cluster state
2. the lifecycle execution info's phase definition is updated from the current
latest policy definition, even though the index isn't moving to a new phase.

Closes #35397.
2018-11-12 11:32:59 -08:00
Nick Knize 2591f66a33
upgrade to lucene-8.0.0-snapshot-6d9c714052 (#35428) 2018-11-12 10:48:27 -06:00
Jason Tedor 3859d21661
Fix the names of CCR stats endpoints in usage API (#35438)
This commit fixes the names of the CCR stats endpoints reported in the
usage API.
2018-11-12 10:27:12 -05:00
Marios Trivyzas 55ff96b43f
SQL: Fix query translation for scripted queries (#35408)
IsNull, IsNotNull and Not where generating wrong queries
as the check to generate ScriptQuery was missing.

Fixes: #35232
2018-11-12 10:42:27 +01:00
Martijn van Groningen ef10461caf
[TEST] Instead of ignoring the ccr downgrade to basic license qa test
avoid the assertions that check the log files, because that does not work on Windows.
The rest of the test is still useful and should work on Windows CI.

Currently on Windows CI this qa module fails because there is just one test and
that test si ignored if OS is Windows.
2018-11-12 10:17:33 +01:00
Yannick Welsch 4e6c58c942 Merge remote-tracking branch 'elastic/master' into zen2 2018-11-12 10:03:59 +01:00
Christoph Büscher 09cac321e7
Upgrade to Joda 2.10.1 (#35410)
This version contains a bugfix that allows us to reenable one of our muted tests
in DateTimeUnitTests.

Closes #33749
2018-11-12 10:02:41 +01:00
Simon Willnauer 64df803af0
Add a java level freeze/unfreeze API (#35353)
This change adds a high level freeze API that allows to mark an
index as frozen and vice versa. Indices must be closed in order to
become frozen and an open but frozen index must be closed to be
defrosted. This change also adds a index.frozen setting to
mark frozen indices and integrates the frozen engine with the
SearchOperationListener that resets and releases the directory
reader after and before search phases.

Relates to #34352
Depends on #34357
2018-11-10 12:15:31 +01:00
Gordon Brown 67f9e8fa23
Enforce limitations on ILM policy names (#35104)
Enforces restrictions on ILM policy names to ensure we don't accept
policy names the system can't handle, or may reserve for future use.
2018-11-09 10:11:26 -07:00
Martijn van Groningen ae2af20ae5
[CCR] Validate remote cluster license as part of put auto follow pattern api call (#35364)
Validate remote cluster license as part of put auto follow pattern api call
in addition of validation that when auto follow coordinator starts auto
following indices in the leader cluster.

Also added qa module that tests what happens to ccr after downgrading to basic license.
Existing active follow indices should remain to follow,
but the auto follow feature should not pickup new leader indices.
2018-11-09 17:43:43 +01:00
Martijn van Groningen 807ce10f73
[TEST] Increased timeout for verifying ccr monitoring. 2018-11-09 15:40:15 +01:00
Martijn van Groningen fba811fa3a
[TEST] increased the number of index and delete ops to make it less likely that all ops exist as soft delete docs. 2018-11-09 15:31:51 +01:00
Martijn van Groningen 83152b3835
[CCR] Get all auto follow patterns and no auto follow metadata (#35381)
Return empty response when querying all auto follow patterns,
but there is no auto follow metadata.
2018-11-09 14:24:27 +01:00
Marios Trivyzas 36da6e1671
SQL: Introduce IsNull node to simplify expressions (#35206)
Add `IsNull` node in parser to simplify expressions so that `<value> IS NULL` is
no longer translated internally to `NOT(<value> IS NOT NULL)`

Replace `IsNotNullProcessor` with `CheckNullProcessor` to encapsulate both
isNull and isNotNull functionality.

Closes: #34876
Fixes: #35171
2018-11-09 11:32:38 +01:00
Armin Braun 529910a43c
DISCOVERY: Fix RollingUpgradeTests (#35375)
* DISCOVERY: Fix RollingUpgradeTests

* Don't manually manage min master nodes if not necessary
* Remove some dead code
* Allow for manually supplying list of seed nodes
* Closes #35178
2018-11-09 10:47:30 +01:00
Albert Zaharovits 617f91bb0f
DOCS Auditing search queries (#35301)
This documents how to include the search queries in the audit log.

There is a catch, that even if enabling `emit_request_body`, which should
output queries included in request bodies, search queries were not output
because, implicitly, no REST layer audit event type was included.

This folk knowledge is herein imprinted.
2018-11-09 11:38:45 +02:00
Tim Vernum a9641960e1
Refactor realm tests to use single settings (#35362)
Many realm tests were written to use separate setting objects for
"global settings" and "realm settings".
Since #30241 there is no distinction between these settings, so these
tests can be cleaned up to use a single Settings object.
2018-11-09 18:13:24 +11:00
Alpar Torok 66e3ca9724 Revert "Mute test for #35361"
This reverts commit f9154758e6.
2018-11-09 08:04:56 +02:00
Marios Trivyzas c9a84fe361
SQL: Handle null literal for AND and OR in `WHERE` (#35236)
Change `nullable()` logic of AND and OR to false since in the Optimizer
we cannot fold to null as we might be handling and expression in the
SELECT clause.

Introduce folding of null for AND and OR expressions in PruneFilter()
since we now know that we are in HAVING or WHERE clause and we
can fold `null` to `false`

Fixes: #35088

Co-authored-by: Costin Leau <costin.leau@gmail.com>
2018-11-08 23:32:59 +01:00
Tim Brooks 93c2c604e5
Move compression config to ConnectionProfile (#35357)
This is related to #34483. It introduces a namespaced setting for
compression that allows users to configure compression on a per remote
cluster basis. The transport.tcp.compress remains as a fallback
setting. If transport.tcp.compress is set to true, then all requests
and responses are compressed. If it is set to false, only requests to
clusters based on the cluster.remote.cluster_name.transport.compress
setting are compressed. However, after this change regardless of any
local settings, responses will be compressed if the request that is
received was compressed.
2018-11-08 10:37:59 -07:00
Jason Tedor 730ec1ddfb
Add more detailed OS name on Linux (#35352)
Today our OS information returned in node stats only returns a
high-level name of the OS (e.g., "Linux"). Yet, for some uses this is
too high-level and knowing at a finer level of granularity the
underlying OS can be useful. This commit extracts the pretty name on
Linux from /etc/os-release. This pretty name usually includes the Linux
vendor and the Linux vendor version number (e.g., Fedora 28).
2018-11-08 12:16:58 -05:00
David Turner 6885a7cb0f
Introduce transport API for cluster bootstrapping (#34961)
- Introduces a transport API for bootstrapping a Zen2 cluster
- Introduces a transport API for requesting the set of nodes that a
  master-eligible node has discovered and for waiting until this comprises the
  expected number of nodes.
- Alters ESIntegTestCase to use these APIs when forming a cluster, rather than
  injecting the initial configuration directly.
2018-11-08 16:09:37 +00:00
Hendrik Muhs 433469d4d0
[Rollup] improve handling of failures on first search (#35269)
Improve error handling in the Indexer if an exception occurs during the very 1st retrieval (query execution)
2018-11-08 16:28:48 +01:00
Martijn van Groningen 07a69a528b
[CCR] Rename leaderClient variables and parameters to remoteClient (#35368) 2018-11-08 16:26:14 +01:00
David Turner 77789a733d Merge branch 'master' into 2018-11-08-merge-master 2018-11-08 13:38:18 +00:00
Albert Zaharovits 025a0c82e5
Remove deprecated audit settings (#35205)
Removes `.prefix` deprecated settings for the logfile
auditing and also documents it in the migrate asciidoc.
2018-11-08 14:06:47 +02:00
David Kyle 9494e046e7
[ML] Prevent notifications on deletion of a non existent job (#35337) 2018-11-08 09:57:07 +00:00
Martijn van Groningen 8a85251da0
[CCR] Auto follow Coordinator fetch cluster state in system context (#35120)
Auto follow Coordinator should fetch the leader cluster state using system context.
2018-11-08 10:48:27 +01:00
Alpar Torok f9154758e6 Mute test for #35361 2018-11-08 11:31:51 +02:00
Tim Vernum 0b24c4f8e7 Test: Mute failing SSL test
Relates: https://github.com/elastic/elasticsearch/issues/35360
2018-11-08 19:37:29 +11:00
Martijn van Groningen 2f2090f562 [CCR] Adjust list of dynamic index settings that should be replicated (#35195)
Adjust list of dynamic index settings that should be replicated
and added a test that verifies whether builtin dynamic index settings
are classified as replicated or non replicated (whitelisted).
2018-11-07 21:59:58 -05:00
Jason Tedor 4f4fc3b8f8
Replicate index settings to followers (#35089)
This commit uses the index settings version so that a follower can
replicate index settings changes as needed from the leader.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2018-11-07 21:20:51 -05:00
Tim Vernum 7d05257896
Rename RealmConfig.globalSettings() to settings() (#35330)
There is no longer a concept of non-global "realm settings". All realm
settings should be loaded from the node's settings using standard
Setting classes.

This change renames the "globalSettings" field and method to simply be
"settings".
2018-11-08 12:57:42 +11:00
Tim Vernum 566979cc31
[TEST] Cleanup FileUserPasswdStoreTests (#35329)
The file realm has not supported custom filenames/locations since at
least 5.0, but this test still tried to configure them.

Remove all configuration of file locations, and cleaned up a few other
warnings and deprecations
2018-11-08 12:57:22 +11:00
Jake Landis d7a4cef483
watcher: Fix integration tests to ensure correct start/stop of Watcher (#35271)
Ensure that Watcher is correctly started and stopped between tests for
SmokeTestWatcherWithSecurityIT,
SmokeTestWatcherWithSecurityClientYamlTestSuiteIT,
SmokeTestWatcherTestSuiteIT, WatcherRestIT,
XDocsClientYamlTestSuiteIT, and XPackRestIT

The change here is to throw an `AssertionError` instead of `break;` to
allow the `assertBusy()` to continue to busy wait until the desired
state is reached.

closes #33291, closes #29877, closes #34462, closes #30705, closes #34448
2018-11-07 15:08:03 -06:00
Lee Hinman b1655c468a
Remove ALL shard check in CheckShrinkReadyStep (#35346)
Since it's still possible to shrink an index when replicas are unassigned, we
should not check that all copies are available when performing the shrink, since
we set the allocation requirement for a single node.

Resolves #35321
2018-11-07 13:40:19 -07:00
Lee Hinman c51dcb1f5d
[ILM] Check shard and relocation status in AllocationRoutedStep (#35316)
* [ILM] Check shard and relocation status in AllocationRoutedStep

This is a follow-up from #35161 where we now check for started and relocating
state in `AllocationRoutedStep`.

Resolves #35258
2018-11-07 13:24:21 -07:00
Simon Willnauer 0cc0fd2d15
Add a frozen engine implementation (#34357)
This change adds a `frozen` engine that allows lazily open a directory reader
on a read-only shard. The engine wraps general purpose searchers in a LazyDirectoryReader
that also allows to release and reset the underlying index readers after any and before
secondary search phases.

Relates to #34352
2018-11-07 20:23:35 +01:00
Martijn van Groningen 314b9ca44c
[CCR] Enforce auto follow pattern name restrictions (#35197)
An auto follow pattern:
* cannot start with `_`
* cannot contain a `,`
* can be encoded in UTF-8
* the length of UTF-8 encoded bytes is no longer than 255 bytes
2018-11-07 20:16:26 +01:00
Tal Levy 4167c14d6c
[ILM] rolling upgrade tests (#35328)
Adds basic rolling upgrade tests to check that lifecycles are still recognizable between rolling cluster upgrades and managed indices stay managed.

This is a placeholder for discussing types of checks so they are ready once we backported

This is a re-boot of the previous PR against index-lifecycle that needed to be
reverted due to CI bwc issues. #32828
2018-11-07 10:50:55 -08:00
Benjamin Trent 2117f4f358
[ML] Add Missing data checking class (#35310)
* ML: Adding missing data check class

* reverting bad change

* Adding bucket + missing data object for returns

* reverting unnecessary change

* adding license header

* Make client calls synchronous, akin to DatafeedJob

* Fixing line length

* Renaming things, addressing PR comments
2018-11-07 12:48:15 -06:00
Martijn van Groningen e685cfe8f9
[CCR] Fail with a better error if leader index is red (#35298)
as part of fetching history uuids from leader index.
2018-11-07 13:23:30 +01:00
Alpar Torok 8a85b2eada
Remove build qualifier from server's Version (#35172)
With this change, `Version` no longer carries information about the qualifier,
we still need a way to show the "display version" that does have both
qualifier and snapshot. This is now stored  by the build and red from `META-INF`.
2018-11-07 14:01:05 +02:00
Marios Trivyzas 42dcdd0aa8
SQL: Improve CircuitBreaker logic for SqlParser (#35300)
Grammar's identifiers can be completely skipped from counting depths
as they just add another level to the tree and they are always children
of some other expression which gets counted.

Increased maximum depth from 100 to 200. After testing on production
configuration with -Xss1m, depths of at least 250 can be used, so being
conservative we put the limit lower.

Fixes: #35299
2018-11-07 12:08:32 +01:00
Alexander Reelsen 9f3effd690
Watcher: Ignore system locale/timezone in croneval CLI tool (#33215)
The elasticsearch-croneval CLI tool uses local dates to display when
something gets triggered the next time. This is very confusing.

This commit ensures, that UTC and local timezone times will be written
out.

The output looks like this and contains localized dates for each trigger
date as well as for `now`.

Now is [Tue, 28 Aug 2018 17:23:51 +0000] in UTC, local time is [ᏔᎵᏁ, 28 ᎦᎶ 2018 12:23:51 -0500]
Here are the next 10 times this cron expression will trigger:
1.	Mon, 2 Jan 2040 11:00:00 +0000
	ᏉᏅᎯ, 2 ᎤᏃ 2040 06:00:00 -0500
2.      ...

This also removes an old outstanding TODO to use the jopt parsing to
cast the count to an integer instead of doing it ourselves.
2018-11-07 09:03:35 +01:00
Tim Vernum b4173c8393
Remove deprecated RealmConfig constructor (#35327)
This removes an obsolete constructor that was still being called from
some tests.

Relates: #30241
2018-11-07 18:21:30 +11:00
Martijn van Groningen 2395e16d84
[CCR] Change resume follow api to be a master node action (#35249)
In order to start shard follow tasks, the resume follow api already
needs execute N requests to the elected master node.

The pause follow API is also a master node action, which would make
how both APIs execute more consistent.
2018-11-07 07:38:44 +01:00
Tim Brooks f395b1eace
Open node connections asynchronously (#35144)
This is related to #29023. Additionally at other points we have
discussed a preference for removing the need to unnecessarily block
threads for opening new node connections. This commit lays the groudwork
for this by opening connections asynchronously at the transport level.
We still block, however, this work will make it possible to eventually
remove all blocking on new connections out of the TransportService
and Transport.
2018-11-06 17:58:20 -07:00
Andrei Stefan 432c518266
Upgrade jline to 3.8.2 (#35288) 2018-11-06 18:42:57 +02:00
Andrei Stefan c3e4575f61
SQL: new SQL CLI logo (#35261)
Added new SQL CLI logo representing the Elastic logo "painted" with
"Elastic" words, "SQL" under the logo and version on the last line
2018-11-06 17:51:52 +02:00
Tal Levy a85b4f42ca
[ILM] change remove-policy-from-index http method from DELETE to POST (#35268)
The remove-ilm-from-index API was using the DELETE http method
to signify that something is being removed. Although, metadata
about ILM for the index is being deleted, no entity/resource
is being deleted during this operation. POST is more in line with
what this API is actually doing, it is modifying the metadata for
an index. As part of this change, `remove` is also appended to the path 
to be more explicit about its actions.
2018-11-06 07:46:25 -08:00
Martijn van Groningen a937d7f5f3
[CCR] Forgot missing return statement,
Error was thrown if leader index had no soft deletes enabled, but it then continued creating the follower index.

The test caught this bug, but very rarely due to timing issue.

Build failure instance:

```
1> [2018-11-05T20:29:38,597][INFO ][o.e.x.c.LocalIndexFollowingIT] [testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes] before test
  1> [2018-11-05T20:29:38,599][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [[]] to [["127.0.0.1:9300"]]
  1> [2018-11-05T20:29:38,599][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [[]] to [["127.0.0.1:9300"]]
  1> [2018-11-05T20:29:38,609][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s_0] [leader-index] creating index, cause [api], templates [random-soft-deletes-templat
e, one_shard_index_template], shards [2]/[0], mappings []
  1> [2018-11-05T20:29:38,628][INFO ][o.e.c.r.a.AllocationService] [node_s_0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[leader-
index][0]] ...]).
  1> [2018-11-05T20:29:38,660][INFO ][o.e.x.c.a.TransportPutFollowAction] [node_s_0] [follower-index] creating index, cause [ccr_create_and_follow], shards [2]/[0]
  1> [2018-11-05T20:29:38,675][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [["127.0.0.1:9300"]] to [[]]
  1> [2018-11-05T20:29:38,676][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [["127.0.0.1:9300"]] to [[]]
  1> [2018-11-05T20:29:38,678][INFO ][o.e.x.c.LocalIndexFollowingIT] [testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes] after test
  1> [2018-11-05T20:29:38,678][INFO ][o.e.x.c.LocalIndexFollowingIT] [testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes] [LocalIndexFollowingIT#testDoNotCreateFoll
owerIfLeaderDoesNotHaveSoftDeletes]: cleaning up after test
  1> [2018-11-05T20:29:38,678][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s_0] [follower-index/TlWlXp0JSVasju2Kr_hksQ] deleting index
  1> [2018-11-05T20:29:38,678][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s_0] [leader-index/FQ6EwIWcRAKD8qvOg2eS8g] deleting index
FAILURE 0.23s J0 | LocalIndexFollowingIT.testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes <<< FAILURES!
   > Throwable #1: java.lang.AssertionError:
   > Expected: <false>
   >      but: was <true>
   >    at __randomizedtesting.SeedInfo.seed([7A3C89DA3BCA17DD:65C26CBF6FEF0B39]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.xpack.ccr.LocalIndexFollowingIT.testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes(LocalIndexFollowingIT.java:83)
   >    at java.lang.Thread.run(Thread.java:748)
```

Build failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+intake/46/console
2018-11-06 16:05:35 +01:00
Marios Trivyzas 1be64a7b8b
SQL: Fix null handling for AND and OR in SELECT (#35277)
Override `process()` in `BinaryLogicProcessor` which doesn't immediately
return null if left or right argument is null, which is the behaviour of
`process()` of the parent class `BinaryProcessor`.

Also, add more tests for `AND` and `OR` in SELECT clause with literal.

Fixes: #35240
2018-11-06 15:40:16 +01:00
Tal Levy 2bf843e768 [TEST] Mute ChangePolicyForIndexIT#testChangePolicyForIndex 2018-11-06 06:09:49 -08:00
Costin Leau 75e9a639ee
SQL: Introduce Coalesce function (#35253)
Add Coalesce conditional for replacing null values

Fix #35060
2018-11-06 13:12:24 +02:00
Nick Knize a5e1f4d3a2 Upgrade to lucene-8.0.0-snapshot-31d7dfe6b1 (#35224) 2018-11-06 11:55:23 +01:00
Simon Willnauer 833e0f8ecf
Prevent throttled indices to be searched through wildcards by default (#34354)
Today if a wildcard, date-math expression or alias expands/resolves
to an index that is search-throttled we still search it. This is likely
not the desired behavior since it can unexpectedly slow down searches
significantly.

This change adds a new indices option that allows `search`, `count`
and `msearch` to ignore throttled indices by default. Users can
force expansion to throttled indices by using `ignore_throttled=true`
on the rest request to expand also to throttled indices.

Relates to #34352
2018-11-06 09:45:30 +01:00
Martijn van Groningen 46c238d792
[CCR] Improve error when operations are missing (#35179)
Improve error when operations are missing
2018-11-06 08:42:47 +01:00
Martijn van Groningen cac67f8bcc
[CCR] Add extra validation to unfollow api (#35245)
Validate whether the follow index actually exists and
whether the follow index actually has custom ccr metadata.
2018-11-06 08:00:34 +01:00
Tim Vernum 574ec6686e
Include realm type in Security Realm setting keys (#30241)
This moves all Realm settings to an Affix definition.
However, because different realm types define different settings
(potentially conflicting settings) this requires that the realm type
become part of the setting key.

Thus, we now need to define realm settings as:

    xpack.security.authc.realms:
      file.file1:
        order: 0

      native.native1:
        order: 1

- This is a breaking change to realm config
- This is also a breaking change to custom security realms (SecurityExtension)
2018-11-06 14:56:50 +11:00
Shaunak Kashyap 00e66bab36
[Elasticsearch Monitoring] Collect only display_name (for now) (#35265)
This is a forward port of some of the changes made in #8445, specifically the change mentioned in https://github.com/elastic/elasticsearch/pull/34023#issuecomment-433212636.

Currently, in master, the `cluster_stats` collector collects _all_ cluster metadata and indexes it into `.monitoring-es-*`. However, per the discussion linked to above, we decided to collect _only_ the `display_name` cluster metadata setting for now. This PR makes this change.
2018-11-05 16:27:05 -08:00
Marios Trivyzas 9ac7af6b3e
SQL: Introduce NotEquals node to simplify expressions (#35234)
Add NotEquals node in parser to simplify expressions so that <value1> != <value2> is
no longer translated internally to NOT(<value1> = <value2>)

Closes: #35210
Fixes: #35233
2018-11-05 22:23:07 +01:00
Nik Everett f72ef9b5fd
Build: Pull "skip assemble on qa" to common build (#35214)
Pull all of the logic that we use to skip the `assemble` and
`dependenciesInfo` tasks on `qa` projects into one spot in our root
build file.
2018-11-05 16:16:00 -05:00
Lee Hinman 3ee004c7b9
Add dedicated step for checking shrink allocation status (#35161)
This adds a new step for checking whether an index is allocated correctly based
on the rules added prior to running the shrink step. It also fixes a bug where
for shrink we are not allowed to have the shards relocating for the shrink step.

This also allows us to simplify AllocationRoutedStep and provide better
feedback in the step info for why either the allocation or the shrink checks
have failed.

Resolves #34938
2018-11-05 12:02:59 -07:00
Marios Trivyzas fd82813660
SQL: [tests] Fix select csv spec and enable it (#35239)
Also, replace `||` and `&&` with `OR` and `AND` as the former are not
SQL standard operators.
2018-11-05 18:45:58 +01:00
Jake Landis c2766b65cf
ingest: raise visibility of ingest plugin documentation (#35048)
* move the set security user processor to the main documentation
* link to plugin processors

part of #33188
2018-11-05 11:44:10 -06:00
Gordon Brown 0fbb8a16bc
Skip Rollover step if next index already exists (#35168)
If the Rollover step would fail due to the next index in sequence
already existing, just skip to the next step instead of going to the
Error step.

This prevents spurious `ResourceAlreadyExistsException`s created by
simultaneous RolloverStep executions from causing ILM to error out
unnecessarily.
2018-11-05 09:20:43 -07:00
Lee Hinman 3473217563
Remove Joda usage from ILM (#35220)
This commit removes the Joda time usage from ILM and the HLRC components of ILM.
It also fixes an issue where using the `?human=true` flag could have caused the
parser not to work. These millisecond fields now follow the standard we use
elsewhere in the code, with additional fields added iff the `human` flag is
specified.

This is a breaking change for ILM, but since ILM has not yet been released, no
compatibility shim is needed.
2018-11-05 08:17:15 -07:00
Alexander Reelsen 409050e8de
Refactor: Remove settings from transport action CTOR (#35208)
As settings are not used in the transport action constructor, this
removes the passing of the settings in all the transport actions.
2018-11-05 13:08:18 +01:00
Martijn van Groningen ddda2d419c
[CCR] Change max_read_request_size default (#35247)
This changes the max_read_request_size default from unlimited to 32MB.
2018-11-05 12:51:42 +01:00
David Kyle 85f8458f06 [ML] Add comment describing test behaviour 2018-11-05 11:21:59 +00:00
Nhat Nguyen 54e1231ebd
CCR/TEST: Limit indexing docs in FollowerFailOverIT (#35228)
The suite FollowerFailOverIT is failing because some documents are not
replicated to the follower. Maybe the FollowTask is not working as
expected or the background indexers eat all resources while the follower
cluster is trying to reform after a failover; then CI is not fast enough
to replicate all the indexed docs within 60 seconds (sometimes I see 80k
docs on the leader).

This commit limits the number of documents to be indexed into the leader
index by the background threads so that we can eliminate the latter
case. This change also replaces a docCount assertion with a docIds
assertion so we can have more information if these tests fail again.

Relates #33337
2018-11-03 10:00:54 -04:00
Jay Modi ec1c2c4a76
Remove AbstractComponent usage in x-pack core (#35187)
This change removes the use of AbstractComponent in the security
module. The classes now declare their own loggers.

Relates #34488
2018-11-02 14:30:43 -06:00
Gordon Brown b3da3eae08
[ILM] Fix race condition in test (#35143)
Previously, testRunStateChangePolicyWithNextStep asserted that the
ClusterState before and after running the steps were equal. The test
only passed due to a race condition: The latch would be triggered by the
step execution, but the cluster state update thread would continue
running before committing the change to the cluster state. This allowed
the test to read the old cluster state and pass the equality check about
99.99% of the time.

The test now waits for the new cluster state to be committed before
checking that it is _not_ equal to the old cluster state.
2018-11-02 11:09:48 -06:00
Jason Tedor 1e241190eb
Disable assemble task from ILM qa projects
This commit disables the assemble tasks from all ILM qa projects. These
projects do not have an assemble task to execute.
2018-11-02 11:16:34 -04:00
David Kyle 43cff56aec Mute FollowerFailOverIT.testFollowIndexAndCloseNode
Issue #33337
2018-11-02 15:13:09 +00:00
Alan Woodward e2af849f70
Move ObjectPath and XContentUtils to libs/x-content (#34803)
These are generally useful utility classes that do not need to live in the Watcher code
2018-11-02 15:12:09 +00:00
Jay Modi 6b08d5fc89
Remove use of AbstractComponent in security (#35186)
This change removes the use of AbstractComponent in the security
module. The classes now declare their own loggers.

Relates #34488
2018-11-02 08:33:26 -06:00
Benjamin Trent 7fb95be8bd
Moving xpack.plugin.core ml response to use new base class (#35157)
* Moving xpack.plugin.core ml response to use new base class

* changing back to streamable
2018-11-02 07:46:53 -05:00
Colin Goodheart-Smithe 0aa66b077a
Merge branch 'master' into index-lifecycle 2018-11-02 11:20:54 +00:00
Nhat Nguyen 4875d6fb0b
CCR: Add NodeClosedException to retryable list (#35191)
This change adds NodeClosedException to the retry-able exception list.
2018-11-02 07:01:46 -04:00
Colin Goodheart-Smithe fc6e1f7f3f
Merge branch 'master' into index-lifecycle 2018-11-02 10:56:35 +00:00
Marios Trivyzas 340363b9fa
SQL: Fix null handling for IN => painless script (#35124)
Include `null` literals when generating the painless script for `IN` expressions.
Previously, they were skipped, because of an issue that had been fixed with #35108.

Fixes: #35122
2018-11-02 10:37:54 +01:00
Alpar Torok b432cbdafe Fix upgrade tests involving snapshot
Fixes
```
Could not determine the dependencies of task ':x-pack:qa:full-cluster-restart:with-system-key:v5.6.13#oldClusterTestCluster#node1.copyBwcPlugins'.
```
2018-11-02 09:51:41 +02:00
Martijn van Groningen 19d6cf1b9e
[CCR] Change response classes to not use from Streamable. (#35085)
Only the response classes of get auto follow pattern, the follow and stats APIs
were moved away from Streamable.  The other APIs use `AcknowledgedResponse`
or `BaseTasksResponse` as response class and
moving that class away from Streamable is a bigger change.
2018-11-02 08:02:17 +01:00
Nhat Nguyen dbc7c9259c CCR/TEST: Add debug log to testFailOverOnFollower 2018-11-01 22:28:06 -04:00
Tal Levy b260406ecf
ILM: use node _id attribute when allocating to one node (#35061)
ILM's Shrink Action was using a nodes "_name" attribute to
allocate to prepare for the shrink step. Since the name is
configurable by a user and may use the same name for
multiple nodes on one machine, _id is safer since it is guaranteed
to be unique.

closes #35043.
2018-11-01 14:38:00 -07:00
Tal Levy 6b312a500d uninherit from AbstractComponent in IndexLifecycleService 2018-11-01 10:22:55 -07:00
Tal Levy c3cf7dd305 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-11-01 10:13:02 -07:00
Alpar Torok f8378d91a6
Implement VersionCollection in Java (#34050) 2018-11-01 17:43:57 +02:00
Benjamin Trent 2fadec5c3d
ML: Add support for rollup Indexes in Datafeeds (#34654)
* Adding rollup support for datafeeds

* Fixing tests and adjusting formatting

* minor formatting chagne

* fixing some syntax and removing redundancies

* Refactoring and fixing failing test

* Refactoring, adding paranoid null check

* Moving rollup into the aggregation package

* making AggregationToJsonProcessor package private again

* Addressing test failure

* Fixing validations, chunking

* Addressing failing test

* rolling back RollupJobCaps changes

* Adding comment and cleaning up test

* Addressing review comments and test failures

* Moving builder logic into separate methods

* Addressing PR comments, adding test for rollup permissions

* Fixing test failure

* Adding rollup priv check on datafeed put

* Handling missing index when getting caps

* Fixing unused import
2018-11-01 10:02:24 -05:00
Igor Motov 7b13d0592f
Watcher: fix metric stats names (#34951)
* Watcher: fix metric stats names

The current watcher stats metric names doesn't match the current
documentation. This commit fixes the behavior of `queued_watches`
metric, deprecates `pending_watches` metric and adds `current_watches`
to match the documented behavior. It also fixes the documentation, which
introduced `executing_watches` metric that was never added.

Fixes #34865
2018-11-01 10:12:05 -04:00