Commit Graph

41431 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe 8e59de3eb2
Merge branch 'master' into index-lifecycle 2018-09-13 09:46:14 +01:00
Marios Trivyzas 7d3b99a9b7
SQL: Fix result column names for CAST (#33604)
Previously, when an non-pruned cast (casting as a different
data type) got applied on a table column in the `SELECT` clause,
the name of the result column didn't contain the target data type
of the cast, e.g.:

  SELECT CAST(MAX(salary) AS DOUBLE) FROM "test_emp"

returned as column name:

  CAST(MAX(salary))

instead of:

  CAST(MAX(salary) AS DOUBLE)

Closes #33571

* Added more tests for trivial casts that are pruned
2018-09-13 10:28:05 +02:00
Jim Ferenczi 6ca36bba15
Fix field mapping updates with similarity (#33634)
This change fixes a bug introduced in 6.3 that prevents fields with an explicit
similarity to be updated. It also adds a test that checks this case for similarities
but also for analyzers since they could suffer from the same problem.

Closes #33611
2018-09-13 09:21:27 +02:00
David Turner 5a3fd8e4e7
Use file-based discovery not MockUncasedHostsProvider (#33554)
Today we use a special unicast hosts provider, the `MockUncasedHostsProvider`,
in many integration tests, to deal with the dynamic nature of the allocation of
ports to nodes. However #33241 allows us to use file-based discovery to achieve
the same goal, so the special test-only `MockUncasedHostsProvider` is no longer
required.

This change removes `MockUncasedHostProvider` and replaces it with file-based
discovery in tests based on `EsIntegTestCase`.
2018-09-13 07:37:15 +02:00
David Turner 94f6d4560d Revert "Mark NamedDateTimeProcessorTests as @AwaitsFix"
This reverts commit 44a80d61b2.
2018-09-13 07:32:33 +02:00
David Turner 44a80d61b2 Mark NamedDateTimeProcessorTests as @AwaitsFix 2018-09-13 07:24:51 +02:00
Nhat Nguyen b097eff342
Resync fails to notify on unavaiable exceptions (#33615)
We fail to notify the resync listener if the resync replication hits a
shard unavailable exception. Moreover, we no longer need to swallow
these unavailable exceptions.

Relates #28571
Closes #33613
2018-09-12 21:27:59 -04:00
Shaunak Kashyap b2f78e5a47
[CCR Monitoring] Only collect stats for specified indices (#33646)
Follow up to #33617. Relates to #30086.

As with all other per-index Monitoring collectors, the `CcrStatsCollector` should only collect stats for the indices the user wants to monitor. This list is controlled by the `xpack.monitoring.collection.indices` setting and defaults to all indices.
2018-09-12 15:56:13 -07:00
Jason Tedor eb715d5290
Add follower index to CCR monitoring and status (#33645)
This commit adds the follower index to CCR shard follow task status, and
to monitoring.
2018-09-12 17:35:06 -04:00
Martijn van Groningen b5d8495789
[CCR] Add auto follow pattern APIs to transport client. (#33629) 2018-09-12 21:50:22 +02:00
Jay Modi 20c6c9c542
Address license state update/read thread safety (#33396)
This change addresses some issues regarding thread safety around
updates and method calls on the XPackLicenseState object. There exists
a possibility that there could be a concurrent update to the
XPackLicenseState when there is a scheduled check to see if the license
is expired and a cluster state update. In order to address this, the
update method now has a synchronized block where member variables are
updated. Each method that reads these variables is now also
synchronized.

Along with the above change, there was a consistency issue around
security calls to the license state. The majority of security checks
make two calls to the license state, which could result in incorrect
behavior due to the checks being made against different license states.
The majority of this behavior was introduced for 6.3 with the inclusion
of x-pack in the default distribution. In order to resolve the majority
of these cases, the `isSecurityEnabled` method is no longer public and
the logic is also included in individual methods about security such as
`isAuthAllowed`. There were a few cases where this did not remove
multiple calls on the license state, so a new method has been added
which creates a copy of the current license state that will not change.
Callers can use this copy of the license state to make decisions based
on a consistent view of the license state.
2018-09-12 13:08:09 -06:00
Jason Tedor 9b8fe85edb
Remove volatile from global checkpoint listeners (#33636)
This field does not need to be volatile because all accesses are done
under a lock. This commit removes the unnecessary volatile modifier from
this field.
2018-09-12 14:38:24 -04:00
Vladimir Dolzhenko cfc20825ab mute NamedDateTimeProcessorTests 2018-09-12 20:34:56 +02:00
Martijn van Groningen 5fa81310cc
[CCR] Added history uuid validation (#33546)
For correctness we need to verify whether the history uuid of the leader
index shards never changes while that index is being followed.

* The history UUIDs are recorded as custom index metadata in the follow index.
* The follow api validates whether the current history UUIDs of the leader
  index shards are the same as the recorded history UUIDs.
  If not the follow api fails.
* While a follow index is following a leader index; shard follow tasks
  on each shard changes api call verify whether their current history uuid
  is the same as the recorded history uuid.

Relates to #30086 

Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co>
2018-09-12 19:42:00 +02:00
Jason Tedor c023f67c5d
Add migration note for remote cluster settings (#33632)
The remote cluster settings search.remote.* have been renamed to
cluster.remote.* and are automatically upgraded in the cluster state on
gateway recovery, and on put. This commit adds a note to the migration
docs for these changes.
2018-09-12 13:37:11 -04:00
Martijn van Groningen 901d8035d9
[CCR] Update es monitoring mapping and (#33635)
* [CCR] Update es monitoring mapping and
change qa tests to query based on leader index.


Co-authored-by: Jason Tedor <jason@tedor.me>
2018-09-12 19:36:17 +02:00
Simon Willnauer c783488e97
Add `_source`-only snapshot repository (#32844)
This change adds a `_source` only snapshot repository that allows to wrap
any existing repository as a _backend_ to snapshot only the `_source` part
including live docs markers. Snapshots taken with the `source` repository
won't include any indices,  doc-values or points. The snapshot will be reduced in size and
functionality such that it requires full re-indexing after it's successfully restored.

The restore process will copy the `_source` data locally starts a special shard and engine
to allow `match_all` scrolls and searches. Any other query, or get call will fail with and unsupported operation exception.  The restored index is also marked as read-only.

This feature aims mainly for disaster recovery use-cases where snapshot size is
a concern or where time to restore is less of an issue.

**NOTE**: The snapshot produced by this repository is still a valid lucene index. This change doesn't allow for any longer retention policies which is out of scope for this change.
2018-09-12 17:47:10 +02:00
Vladimir Dolzhenko 141c6ef93e
upgrade randomizedrunner to 2.7.0 (#33623) 2018-09-12 17:18:34 +02:00
Christoph Büscher fe478c23b7
[Docs] Fix heading in composite-aggregation.asciidoc (#33627)
The heading for the "Missing buckets" should be on the same level 
as the the "Order" section.
2018-09-12 16:56:03 +02:00
Jason Tedor 36ba3cda7e
Enable global checkpoint listeners to timeout (#33620)
In cross-cluster replication, we will use global checkpoint listeners to
long poll for updates to a shard. However, we do not want these polls to
wait indefinitely as it could be difficult to discern if the listener is
still waiting for updates versus something has gone horribly wrong and
cross-cluster replication is stuck. Instead, we want these listeners to
timeout after some period (for example, one minute) so that they are
notified and we can update status on the following side that
cross-cluster replication is still active. After this, we will
immediately enter back into a poll mode.

To do this, we need the ability to associate a timeout with a global
checkpoint listener. This commit adds this capability.
2018-09-12 10:53:22 -04:00
Tanguy Leroux bcac7f5e55 Fix checkstyle violation in ShardFollowNodeTask 2018-09-12 16:03:52 +02:00
Dimitris Athanasiou 2eb2313b60
[HLRC][ML] Add ML put datafeed API to HLRC (#33603)
This also changes both `DatafeedConfig` and `DatafeedUpdate`
to store the query and aggs as a bytes reference. This allows
the client to remove its dependency to the named objects
registry of the search module.

Relates #29827
2018-09-12 14:52:36 +01:00
Tanguy Leroux 7e195c2912
Update AWS SDK to 1.11.406 in repository-s3 (#30723) 2018-09-12 15:27:57 +02:00
Jason Tedor 23f12e42c1
Expose CCR stats to monitoring (#33617)
This commit exposes the CCR stats endpoint to monitoring collection.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2018-09-12 09:13:07 -04:00
Joel Green 0b567c0eeb [Docs] Update match-query.asciidoc (#33610) 2018-09-12 14:35:27 +02:00
Nhat Nguyen d9bbb89b26 TEST: Adjust rollback condition when shard is empty
If a shard is empty, it won't rollback its engine on promotion.
This commit adjusts the expectation in the rollback test.

Relates #33473
2018-09-12 08:26:02 -04:00
Martijn van Groningen 96c49e5ed0
[CCR] Improve shard follow task's retryable error handling (#33371)
Improve failure handling of retryable errors by retrying remote calls in
a exponential backoff like manner. The delay between a retry would not be
longer than the configured max retry delay. Also retryable errors will be
retried indefinitely.

Relates to #30086
2018-09-12 12:49:51 +02:00
lipsill c92ec1c5d7 Forbid negative `weight` in Function Score Query (#33390)
This change forbids negative `weight` in Function Score query. Negative scores are forbidden in Lucene 8.
2018-09-12 09:16:40 +02:00
Jim Ferenczi 4561c5ee83
Clarify context suggestions filtering and boosting (#33601)
This change clarifies the documentation of the context completion suggester
regarding filtering and boosting with contexts.
Unlike the suggester v1, filtering on multiple contexts
works as a disjunction, a suggestion matches if it contains at least one of the provided
context values and boosting selects the maximum score among the matching contexts.
This commit also adapts an old test that was written for the v1 suggester and commented out
for version 2 because the behavior changed.
2018-09-12 08:47:32 +02:00
Jason Tedor 20476b9e06
Disable CCR REST endpoints if CCR disabled (#33619)
This commit avoids enabling the CCR REST endpoints if CCR is disabled.
2018-09-12 01:54:34 -04:00
Jason Tedor dcdacd2f3f
Lower version on full cluster restart settings test
The change to upgrade cross-cluster search settings was backported to
6.5.0. Therefore, this assumption needs to be reduced to the latest
version where settings were not automatically upgraded.
2018-09-12 01:19:05 -04:00
Jason Tedor c74c46edc3
Upgrade remote cluster settings (#33537)
This commit adds settings upgraders for the search.remote.* settings
that can be in the cluster state to automatically upgrade these settings
to cluster.remote.*. Because of the infrastructure that we have here,
these settings can be upgraded when recovering the cluster state, but
also when a user tries to make a dynamic update for these settings.
2018-09-12 01:14:43 -04:00
Armin Braun 94cdf0ceba
NETWORKING: http.publish_host Should Contain CNAME (#32806)
* NETWORKING: http.publish_host Should Contain CNAME

* Closes #22029
2018-09-12 06:15:36 +02:00
Jason Tedor 9752540866
Add test coverage for global checkpoint listeners
This commit adds test coverage for two cases not previously covered by
the existing testing. Namely, we add coverage ensuring that the executor
is used to notify listeners being added that are immediately notified
because the shard is closed or because the global checkpoint is already
beyond what the listener knows.
2018-09-11 23:19:27 -04:00
Nhat Nguyen 743327efc2
Reset replica engine to global checkpoint on promotion (#33473)
When a replica starts following a newly promoted primary, it may have
some operations which don't exist on the new primary. Thus we need to
throw those operations to align a replica with the new primary. This can
be done by first resetting an engine from the safe commit, then replaying
the local translog up to the global checkpoint.

Relates #32867
2018-09-11 22:09:37 -04:00
Benjamin Trent 27e07ec859
HLRC: ML Delete Forecast API (#33526)
* HLRC: ML Delete Forecast API
2018-09-11 16:32:52 -07:00
Jason Tedor 9f8dff9281
Remove debug logging in full cluster restart tests (#33612)
These logs are incredibly verbose, and it makes chasing normal failures
burdensome. This commit removes the debug logging, which can be
reenabled again if needed.
2018-09-11 19:11:30 -04:00
Jason Tedor eca37e6e0a
Expose CCR to the transport client (#33608)
This commit exposes CCR to the transport client.
2018-09-11 16:37:52 -04:00
Nhat Nguyen 1e577d3ce8 Mute testIndexDeletionWhenNodeRejoins
Tracked at #33613
2018-09-11 16:23:12 -04:00
Costin Leau 91bca174f5
SQL: Make Literal a NamedExpression (#33583)
* SQL: Make Literal a NamedExpression

Literal now is a NamedExpression reducing the need for Aliases for
folded expressions leading to simpler optimization rules.

Fix #33523
2018-09-11 18:47:49 +03:00
Lisa Cawley cbc6fa0ecb
[DOCS] Adds missing built-in user information (#33585) 2018-09-11 07:56:26 -07:00
Colin Goodheart-Smithe 624b84f897
Improves doc values format deprecation message (#33576)
* Improves doc values format deprecation message

This changes the deprecation message when doc values fields do not
supply a format form logging a deprecation warning for each offending
field individually to logging a single message which lists all
offending fields

Closes #33572

* Updates YAML test with new deprecation message

Also adds a test to ensure multiple deprecation warnings are collated
into one message

* Condenses collection of fields without format check

Moves the collection of fields that don't have a format to a separate
loop and moves the logging of the deprecation warning to be next to it
at the expesnse of looping through the field list twice

* fixes typo

* Fixes test
2018-09-11 14:32:43 +01:00
Jason Tedor 73c75bef21
Preserve cluster settings on full restart tests (#33590)
Today the full cluster restart tests do not preserve cluster settings on
restart. This is a mistake because it is not an accurate reflection of
reality, we do not expect users to clear cluster settings when they
perform a full cluster restart. This commit makes it so that all full
cluster restart tests preserve settings on upgrade.
2018-09-11 08:40:22 -04:00
Alan Woodward 36bdad4895
Use IndexWriter.getFlushingBytes() rather than tracking it ourselves (#33582)
Currently we keep track of how many bytes are currently being written to disk
in an AtomicLong within InternalEngine, updating it on refresh. The IndexWriter
has its own accounting for this, and exposes it via a getFlushingBytes method
in the latest lucene 8 snapshot. This commit removes the InternalEngine tracking
in favour of just using the IndexWriter method.
2018-09-11 13:38:44 +01:00
Jason Tedor ad4b5e4270
Fix upgrading of list settings (#33589)
Upgrading list settings is broken because of the conversion that we do
to strings, and then when we try to put back the upgraded value we do
not know that it is a representation of a list. This commit addresses
this by adding special handling for list settings.
2018-09-11 08:35:42 -04:00
Simon Willnauer 517cfc3cc0
Add read-only Engine (#33563)
This change adds an engine implementation that opens a reader on an
existing index but doesn't permit any refreshes or modifications
to the index.

Relates to #32867
Relates to #32844
2018-09-11 14:05:14 +02:00
Ed Savage 2f3b542d57
HLRC: Add ML get categories API (#33465)
HLRC: Adding the ML 'get categories' API
2018-09-11 12:48:14 +01:00
Andrei Stefan a3e1f1e46f
SQL: Adds MONTHNAME, DAYNAME and QUARTER functions (#33411)
* Added monthname, dayname and quarter functions
* Updated docs tests with the new functions
2018-09-11 14:35:34 +03:00
Colin Goodheart-Smithe 5a349ec0e6
Renames lifecycle to policy in rest-api-spec (#33507)
* Renames lifecycle to policy in rest-api-spec

* Fix test
2018-09-11 10:18:24 +01:00
Alan Woodward f598297f55
Add predicate_token_filter (#33431)
This allows users to filter out tokens from a TokenStream using painless scripts, 
instead of having to write specialised Java code and packaging it up into a plugin.

The commit also refactors the AnalysisPredicateScript.Token class so that it wraps
and makes read-only an AttributeSource.
2018-09-11 09:16:39 +01:00