Commit Graph

4424 Commits

Author SHA1 Message Date
Jim Ferenczi 12cfdc24b0 Fixed rewrite of time zone without DST (#54398)
We try to rewrite time zones to fixed offsets in the date histogram aggregation
if the data in the shard is within a single transition.
However this optimization is not applied on time zones that don't apply daylight saving changes
but had some random transitions in the past (e.g. Australia/Brisbane or Asia/Katmandu).
This changes fixes the rewrite of such time zones to fixed offsets.
2020-03-30 13:18:57 +02:00
Martijn van Groningen 4b4fbc160d
Refactor AliasOrIndex abstraction. (#54394)
Backport of #53982

In order to prepare the `AliasOrIndex` abstraction for the introduction of data streams,
the abstraction needs to be made more flexible, because currently it really can be only
an alias or an index.

* Renamed `AliasOrIndex` to `IndexAbstraction`.
* Introduced a `IndexAbstraction.Type` enum to indicate what a `IndexAbstraction` instance is.
* Replaced the `isAlias()` method that returns a boolean with the `getType()` method that returns the new Type enum.
* Moved `getWriteIndex()` up from the `IndexAbstraction.Alias` to the `IndexAbstraction` interface.
* Moved `getAliasName()` up from the `IndexAbstraction.Alias` to the `IndexAbstraction` interface and renamed it to `getName()`.
* Removed unnecessary casting to `IndexAbstraction.Alias` by just checking the `getType()` method.

Relates to #53100
2020-03-30 10:12:16 +02:00
Nhat Nguyen 6e025c12f0 Add debug logging for testRunningTasksCount
Relates #53594
2020-03-29 18:34:41 -04:00
Jason Tedor f0033783db
Deprecate node local storage setting (#54374)
This setting is not documented and has dubious value since it means
there can be nodes in the cluster (non-data and non-master nodes) that
do not have persistent node IDs. This does not have any use cases so
this commit removes the setting.
2020-03-28 14:36:41 -04:00
Jason Tedor 60437b474d
Fix line-length violation in DiscoveryNodeRole
This commit fixes a line-length checkstyle violation in
DiscoveryNodeRole.java.
2020-03-28 13:06:20 -04:00
Jason Tedor 03cab96b2d
Fix imports in discovery node classes
This commit fixes some imports that were leftover after resolving some
merge conflicts on a backport.
2020-03-28 12:56:22 -04:00
Jason Tedor c3be3206ce
Decouple environment from DiscoveryNode (#54373)
Today Environment is coupled to DiscoveryNode via the node.local_storage
setting. This commit decouples Environment from this setting.
2020-03-28 12:52:47 -04:00
Jason Tedor 37b59a357f
Ensure that the output of node roles are sorted (#54376)
This commit ensures that node roles are sorted by node role name, which
makes the output easier to consume, and also makes it easier to rely on
the behavior of the output in assertions.
2020-03-28 12:51:21 -04:00
Tim Brooks 2ccddbfa88
Move transport decoding and aggregation to server (#54360)
Currently all of our transport protocol decoding and aggregation occurs
in the individual transport modules. This means that each implementation
(test, netty, nio) must implement this logic. Additionally, it means
that the entire message has been read from the network before the server
package receives it.

This commit creates a pipeline in server which can be passed arbitrary
bytes to handle. Internally, the pipeline will decode, decompress, and
aggregate the messages. Additionally, this allows us to run many
megabytes of bytes through the pipeline in tests to ensure that the
logic works.

This work will enable future work:

Circuit breaking or backoff logic based on message type and byte
in the content aggregator.
Sharing bytes with the application layer using the ref counted
releasable network bytes.
Improved network monitoring based specifically on channels.
Finally, this fixes the bug where we do not circuit break on the correct
message size when compression is enabled.
2020-03-27 14:13:10 -06:00
Stuart Tettemer 1630de4a42
Scripting: stats per context in nodes stats (#54008) (#54357)
Adds script cache stats to `_node/stats`.
If using the general cache:
```
      "script_cache": {
        "sum": {
          "compilations": 12,
          "cache_evictions": 9,
          "compilation_limit_triggered": 5
        }
      }

```
If using context caches:
```
      "script_cache": {
        "sum": {
          "compilations": 13,
          "cache_evictions": 9,
          "compilation_limit_triggered": 5
        },
        "contexts": [
          {
            "context": "aggregation_selector",
            "compilations": 8,
            "cache_evictions": 6,
            "compilation_limit_triggered": 3
          },
          {
            "context": "aggs",
            "compilations": 5,
            "cache_evictions": 3,
            "compilation_limit_triggered": 2
          },
```
Backport of: 32f46f2
Refs: #50152
2020-03-27 12:26:00 -06:00
Tim Brooks f5b4020819
Remove netty BytesReference implementations (#54355)
Elasticsearch has a number of different BytesReference implementations.
These implementations can all implement the interface in different ways
with subtly different behavior and performance characteristics. On the
other-hand, the JVM only represents bytes as an array or a direct byte
buffer. This commit deletes the specialized Netty implementations and
moves to using a generic ByteBuffer reference type. This will allow us
to focus on standardizing performance and behave around a smaller number
of implementations that can be used by all components in Elasticsearch.
2020-03-27 11:01:33 -06:00
Lee Hinman f2cc2b1127
[7.x] Add REST APIs for IndexTemplateV2Metadata CRUD (#54039) (#54347)
* Add REST APIs for IndexTemplateV2Metadata CRUD (#54039)

* Add REST APIs for IndexTemplateV2Metadata CRUD

This commit adds the get/put/delete APIs for interacting with the now v2 versions of index
templates.

These APIs are behind the existing `es.itv2_feature_flag_registered` system property feature flag.

Relates to #53101

* Add exceptions for HLRC tests

* Add skips for 7.x versions

* Use index_template instead of template_v2 in action names

* Add test for MetaDataIndexTemplateService.addIndexTemplateV2

* Move removal to static method and add test

* Add unit tests for request classes (implement hashCode & equals)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* Fix compilation

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-27 10:47:22 -06:00
Dan Hermann 1690e78646
Validation for data stream creation 2020-03-27 10:07:46 -05:00
Alan Woodward 461f1307d6 Add XContentHelper.childBytes() method (#54287)
We have a number of places where we want to read a fairly complex object from
XContent, but aren't interested in its contents; for example, mappings are often
serialized and deserialized between several objects before they are actually built
into a MappingMetaData object. This means that potentially large maps of maps
are constructed several times, only to immediately be re-serialized again.

This commit adds a new helper method to XContentHelper that reads the children
of an xcontent object directly to a BytesReference, serialized via the same xcontenttype
as the parent parser, avoiding the construction of intermediary maps or lists.
2020-03-27 14:21:56 +00:00
Armin Braun 14b5daad7c
Fix Snapshot Completion Listener Lost on Master Failover (#54286) (#54330)
* Fix Snapshot Completion Listener Lost on Master Failover

If master fails over before (or we run into any other exception) when removing
the snapshot from the CS we must still resolve all the completion listeners for
the snapshot.
2020-03-27 14:11:13 +01:00
Gordon Brown 0d30b48613
Disallow negative TimeValues (#53913)
This commit causes negative TimeValues, other than -1 which is sometimes used as
a sentinel value, to be rejected during parsing.

Also introduces a hack to allow ILM to load policies which were written to the
cluster state with a negative min_age, treating those values as 0, which should
match the behavior of prior versions.
2020-03-26 13:30:35 -06:00
William Brafford 14204f8381
Use set-based interface for NodesStatsRequest (#53637) (#54141)
The NodesStatsRequest class uses a set of strings for its internal
serialization. This commit updates the class's interface so that we
no longer use hard-coded getters and setters, but rather
methods that add strings directly. For example, the old way of
adding "os" metrics to a request would be to call request.os(true).
The new way of doing this is to call request.addMetric("os").

For the time being, the canonical list of metrics is an enum in
NodesStatsRequest. This will eventually be replaced with something
pluggable.
2020-03-26 14:41:49 -04:00
Christoph Büscher da404bbce2
HLRC: Don't send defaults for SubmitAsyncSearchRequest (#54200) (#54266)
Currently we set the defaults for ccsMinimizeRoundtrips, preFilterShardSize and
requestCache on the HLRC SubmitAsyncSearchRequest in the constructor. This is no
longer needed since we now only send the parameters along with the rest request
that are supported (omitting e.g. ccsMinimizeRoundtrips) and the correct
defaults are set on the client side. This change removes setting and sending
these defaults where possible, leaving only the overwrite of batchedReduceSize
with a default value of 5, since the default used in the vanilla SearchRequest
is 512. However, we don't need to send this value along as a request parameter
if its the default since the correct one will be set on the receiving end if no
value is specified.
Also adding tests for RestSubmitAsyncSearchAction that check the correct
defaults are set when parameters are missing on the server side.

Backport of #54200
2020-03-26 19:01:17 +01:00
Nik Everett b6a8de0d89
Fix compilation in Eclipse (backport of #54275) (#54284)
These mock calls cause Eclipse to think that `Exception` can be thrown
because `CheckedFunction`'s lower bound is `Exception`. This makes
Eclipse happy.
2020-03-26 12:53:13 -04:00
Lee Hinman 3e1fc8a5c9
[7.x] #32478 fixed cluster stats return 8EB (#32480) (32972311) (#54273)
Co-authored-by: Darby.Han <darby.han@navercorp.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: 한우람 <hgword@gmail.com>
Co-authored-by: Darby.Han <darby.han@navercorp.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-26 09:12:09 -06:00
Henning Andersen 5bfaa20dd4
Rollover: refactor out cluster state update (#53965) (#54269)
Make it possible to reuse the cluster state update of rollover for
simulation purposes by extracting it. Also now run the full rollover in
the pre-rollover phase and the actual rollover phase, allowing a
dedicated exception in case of concurrent rollovers as well as a more
thorough pre-check.
2020-03-26 16:06:13 +01:00
Yannick Welsch 8176f1c7f8
Delegate toXContent logic from ClusterState to its member classes (#54192)
Backport of #52743

Co-authored-by: Yannick Welsch <yannick@welsch.lu>
2020-03-26 15:08:30 +01:00
Alan Woodward 048797389a Explicitly set mappings in SearchAfterIT (#54262)
We have some very occasional failures in SearchAfterIT, where a search throws
an exception because a shard does not have the mapping for the requested sort
field. The field should have been added in a dynamic mapping update after an
index event, but it seems that there can sometimes be a small delay in propagating
this update to the shards.

This commit changes the test to explicitly define the relevant field at index creation
time.

Fixes #51900
2020-03-26 13:05:19 +00:00
Jason Tedor d8f745736b
Clarify the remove keystore command can handle many (#54244)
The remove keystore command can handle multiple settings. In a few
places, we were not consistent about mentioning this. This commit
addreses this, in the CLI help, and the docs.
2020-03-26 08:49:43 -04:00
William Brafford b11960e3e6
Create set-based interface for NodesInfoRequest (#53410) (#54223)
This commit begins the work of removing the "hard-coded" metric getters
and setters from the NodesInfoRequest classes. We start by providing new
flexible getters and setters. We then update the test classes to remove
the old getters, and then remove those getters.
2020-03-26 07:28:09 -04:00
Yannick Welsch 1ba6783780 Schedule commands in current thread context (#54187)
Changes ThreadPool's schedule method to run the schedule task in the context of the thread
that scheduled the task.

This is the more sensible default for this method, and eliminates a range of bugs where the
current thread context is mistakenly dropped.

Closes #17143
2020-03-26 10:07:59 +01:00
Jason Tedor 6af89e62d1
Allow keystore add-file to handle multiple settings (#54240)
Today the keystore add-file command can only handle adding a single
setting/file pair in a single invocation. This incurs the startup costs
of the JVM many times, which in some environments can be expensive. This
commit teaches the add-file keystore command to accept adding multiple
settings in a single invocation.
2020-03-26 00:07:05 -04:00
Jason Tedor fe8257d981
Allow keystore add to handle multiple settings (#54229)
Today the keystore add command can only handle adding a single
setting/value pair in a single invocation. This incurs the startup costs
of the JVM many times, which in some environments can be expensive. This
commit teaches the add keystore command to accept adding multiple
settings in a single invocation.
2020-03-25 22:58:20 -04:00
Nik Everett 8f40f1435a
Save a little space in agg tree (backport of #53730) (#54213)
This drop the "top level" pipeline aggregators from the aggregation
result tree which should save a little memory and a few serialization
bytes. Perhaps more imporantly, this provides a mechanism by which we
can remove *all* pipelines from the aggregation result tree. This will
save quite a bit of space when pipelines are deep in the tree.

Sadly, doing this isn't simple because of backwards compatibility. Nodes
before 7.7.0 *need* those pipelines. We provide them by setting passing
a `Supplier<PipelineTree>` into the root of the aggregation tree that we
only call if we need to serialize to a version before 7.7.0.

This solution works for cross cluster search because we always reduce
the aggregations in each remote cluster and then forward them back to
the coordinating node. Its quite possible that the coordinating node
needs the pipeline (say it is version 7.1.0) and the gateway node in the
remote cluster doesn't (version 7.7.0). In that case the data nodes
won't send the pipeline aggregations back to the gateway node.
Critically, the gateway node *will* send the pipeline aggregations back
to the coordinating node. This is all managed with that
`Supplier<PipelineTree>`, but *how* it is managed is a bit tricky.
2020-03-25 15:51:16 -04:00
Martijn van Groningen 66861a82a1
Data stream should refer to the backing indices using the Index class (#54199)
Backport of #54189
2020-03-25 20:18:15 +01:00
Bogdan Pintea 77da9dd040 Add version 7.8.0
Add version 7.8.0
2020-03-25 18:10:30 +01:00
jimczi 04fabead14 Fix small typo in SearchService#executeQueryPhase
Relates #54044
2020-03-25 16:16:30 +01:00
Armin Braun 32fa90c9ba
Fix ClusterHealthIT.testHealthOnMasterFailover (#54170) (#54177)
We can run into a state where there's no more events to wait for temporarily
but the cluster still isn't green. I added the wait for green flag to the request
so the assertion for green cluster health below doesn't fail.

Closes #53457
2020-03-25 14:33:31 +01:00
Armin Braun 0a70250201
Fix BlobStoreIncrementalityIT Assertion (#54149) (#54155)
We are using this assertion for identical shard snapshots
for situations where `snapshot1` wasn't the first snapshot
for the tested shard. Hence, we can't assume that it will
not share any files with previous snapshots.
This showed up in failing tests when `snapshot1` was equivalent
to a previous snapshot because no documents were deleted from the
repo randomly in the failing test but even if documents are deleted
there is no guarantee that no files will be shared.

=> I removed this assertion since its immaterial for what is tested
here anyway.

Closes #54034
2020-03-25 12:13:52 +01:00
jimczi e380a5a8c3 Fix off-by one error in TransportSearchActionTests
Closes #54156
2020-03-25 11:41:02 +01:00
Tanguy Leroux b6e482295b
Mute TransportSearchActionTests.testShouldPreFilterSearchShards (#54158)
Relates #53873
Relates #54156
2020-03-25 11:25:06 +01:00
Jim Ferenczi 3b4751bdb7
Avoid I/O operations when rewriting shard search request (#54044) (#54139)
This commit ensures that we rewrite the shard request with a short-lived can_match searcher.
This is required for frozen indices since the high level rewrite is now performed on a network thread where we don't want to perform I/O.

Closes #53985
2020-03-25 09:02:36 +01:00
Jason Tedor 381d7586e4
Introduce formal role for remote cluster client (#54138)
This commit introduce a formal role for identifying nodes that are
capable of making connections to remote clusters.

Relates #53924
2020-03-24 21:59:43 -04:00
Henning Andersen 7ce7aff66e Reindex negative TimeValue fix (#54057)
Reindex would use timeValueNanos(System.nanoTime()). The intended use
for TimeValue is as a duration, not as absolute time. In particular,
this could result in negative TimeValue's, being unsupported in #53913.
Modified to use the bare long nano-second value.
2020-03-24 22:29:09 +01:00
Przemko Robakowski fc498f625a
[7.x] Add validation for component templates (#54023) (#54118)
* Add validation for component templates (#54023)

* Add validation for component templates

This change adds validation to make sure that settings and mappings are correct in
component template. It's done the same way as in index templates - code is reused.

Reletes to #53101

* Fix checkstyle violation

* Update server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

* Update server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateServiceTests.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

* Update server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateServiceTests.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

* Update server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateServiceTests.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

* Update server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Adjusted to 7.7

* unused import fixed

* npe fixeD

* change exception type

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
2020-03-24 22:20:34 +01:00
Jim Ferenczi 025857d949 Fix ShardSearchRequest cache key (#54071)
This commit ensures that we don't use the non-deterministic canReturnNullResponseIfMatchNoDocs
boolean in the cache key of the ShardSearchRequest. The value of this boolean has no influence
on the cacheability of the request.

Closes #32827
2020-03-24 22:15:52 +01:00
Przemko Robakowski 5594d57727
/_cat/shards support path stats (#53461) (#54119)
* _cat/shards support path stats

* fix some style case

* fix some style case

* fix rest-api-spec cat.shards error

* fix rest-api-spec cat.shards bwc error

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: weizijun <weizijun1989@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-24 20:46:05 +01:00
Tim Brooks b21b7fb09b
Allow proxy mode server name to be updated (#54107)
Currently there is a bug where the proxy strategy will not be rebuilt if
the server_name is dynamically updated. This commit fixes this issue.
2020-03-24 11:54:23 -06:00
David Turner 21afc788f8 Reduce log level for pipeline failure (#54097)
Today we log `failed to execute pipeline for a bulk request` at `ERROR` level
if an attempt to run an ingest pipeline fails. A failure here is commonly due
to an `EsRejectedExecutionException`. We also feed such failures back to the
client and record the rejection in the threadpool statistics.

In line with #51459 there is no need to log failures within actions so noisily
and with such urgency. It is better to leave it up to the client to react
accordingly. Typically an `EsRejectedExecutionException` should result in the
client backing off and retrying, so a failure here is not normally fatal enough
to justify an `ERROR` log at all.

This commit reduces the log level for this message to `DEBUG`.
2020-03-24 17:41:35 +00:00
markharwood 6a60f85bba
Wildcard field - add normalizer support (#53851) (#54109)
Backport support for normalisation to wildcard field

Closes #53603
2020-03-24 17:37:47 +00:00
Yannick Welsch e006d1f6cf Use special XContent registry for node tool (#54050)
Fixes an issue where the elasticsearch-node command-line tools would not work correctly
because PersistentTasksCustomMetaData contains named XContent from plugins. This PR
makes it so that the parsing for all custom metadata is skipped, even if the core system would
know how to handle it.

Closes #53549
2020-03-24 17:40:51 +01:00
Nik Everett 42be39177b
Remove ceremony declaring aggs (backport of #53990) (#54099)
This removes some more ceremony when declaring agg parsers. You no
longer need a static `parse` method, instead you can just make the
`PARSER` public in most cases.

There are still a few aggs with the `parse` method, but those `parse`
methods are a little more complex to untangle.
2020-03-24 12:29:52 -04:00
Tim Brooks caefa78513
Align remote info api with new settings (#54102)
Currently the remote info api has added a number of possible fields
(proxy, num_socket_connections, etc) that are available in proxy mode.
These fields are not aligned with what the settings are named. This
commit modifies this API to align with the settings.
2020-03-24 10:27:24 -06:00
Christoph Büscher 1c1730facd
Mask wildcard query special characters on keyword queries (#53127) (#53512)
Wildcard queries on keyword fields get normalized, however this normalization
step should exclude the two special characters * and ? in order to keep the
wildcard query itself intact.

Closes #46300
2020-03-24 17:22:29 +01:00
Alan Woodward 39d7d0dc10 Upgrade to lucene 8.5.0 release (#54077)
Upgrades our lucene dependency to the released 8.5.0 version.
2020-03-24 13:45:50 +00:00