Commit Graph

4644 Commits

Author SHA1 Message Date
Martijn van Groningen 6d03081560
Add auto create action (#56122)
Backport of #55858 to 7.x branch.

Currently the TransportBulkAction detects whether an index is missing and
then decides whether it should be auto created. The coordination of the
index creation also happens in the TransportBulkAction on the coordinating node.

This change adds a new transport action that the TransportBulkAction delegates to
if missing indices need to be created. The reasons for this change:

* Auto creation of data streams can't occur on the coordinating node.
Based on the index template (v2) either a regular index or a data stream should be created.
However if the coordinating node is slow in processing cluster state updates then it may be
unaware of the existence of certain index templates, which then can load to the
TransportBulkAction creating an index instead of a data stream. Therefor the coordination of
creating an index or data stream should occur on the master node. See #55377

* From a security perspective it is useful to know whether index creation originates from the
create index api or from auto creating a new index via the bulk or index api. For example
a user would be allowed to auto create an index, but not to use the create index api. The
auto create action will allow security to distinguish these two different patterns of
index creation.
This change adds the following new transport actions:

AutoCreateAction, the TransportBulkAction redirects to this action and this action will actually create the index (instead of the TransportCreateIndexAction). Later via #55377, can improve the AutoCreateAction to also determine whether an index or data stream should be created.

The create_index index privilege is also modified, so that if this permission is granted then a user is also allowed to auto create indices. This change does not yet add an auto_create index privilege. A future change can introduce this new index privilege or modify an existing index / write index privilege.

Relates to #53100
2020-05-04 19:10:09 +02:00
Julie Tibshirani 6b5cf1b031 For constant_keyword, make sure exists query handles missing values. (#55757)
It's possible for a constant_keyword to have a 'null' value before any documents
are seen that contain a value for the field. In this case, no documents have a
value for the field, and 'exists' queries should return no documents.
2020-05-04 09:41:52 -07:00
Armin Braun e8ef44ce78
Allow Bulk Snapshot Deletes to Abort (#56009) (#56111)
Making use of #55773 to simplify snapshot state machine.
1. Deletes with no in-progress snapshot now add the delete entry to the cluster state right away
instead of doing a second CS update after the fist update was a NOOP.
2. If a bulk delete matches in-progress as well as completed snapshots, abort the in-progress snapshot
and then move on to delete from the repository.
2020-05-04 16:21:00 +02:00
Christos Soulios c65f828cb7
[7.x] Histogram field type support for ValueCount and Avg aggregations (#56099)
Backports #55933 to 7.x

Implements value_count and avg aggregations over Histogram fields as discussed in #53285

- value_count returns the sum of all counts array of the histograms
- avg computes a weighted average of the values array of the histogram by multiplying each value with its associated element in the counts array
2020-05-04 13:23:02 +03:00
Armin Braun e01b999ef0
Add Functionality to Consistently Read RepositoryData For CS Updates (#55773) (#56091)
Using optimistic locking, add the ability to run a repository state
update task with a consistent view of the current repository data.
Allows for a follow-up to remove the snapshot INIT state.
2020-05-04 08:13:14 +02:00
Armin Braun 3a64ecb6bf
Allow Deleting Multiple Snapshots at Once (#55474) (#56083)
* Allow Deleting Multiple Snapshots at Once (#55474)

Adds deleting multiple snapshots in one go without significantly changing the mechanics of snapshot deletes otherwise.
This change does not yet allow mixing snapshot delete and abort. Abort is still only allowed for a single snapshot delete by exact name.
2020-05-03 20:30:58 +02:00
David Turner 69f50fe79f Improve same-shard allocation explanations (#56010)
I see occasional confusion about the explanations emitted by the same-shard
allocation decider, particularly amongst new users setting up a single-node
cluster and trying to determine why their cluster has `yellow` health. For
example:

    the shard cannot be allocated to the same node on which a copy of the shard
    already exists

This is technically correct but it's quite a complicated sentence. Also, by
starting with "the shard cannot be allocated" it makes it sound like this is
the problem, whereas in fact this message is a good thing and users should
typically focus their attention elsewhere.

This commit simplifies the wording of these messages and makes them sound more
positive, for example:

    a copy of this shard is already allocated to this node
2020-05-01 10:07:14 +01:00
Mark Tozzi d8eb51ed63
Wire up GeoDistanceAggregation (#55975) (#56042) 2020-04-30 15:43:27 -04:00
Tim Brooks 54dbea6c65
Improve RemoteConnectionManager consistency (#55759)
In order to iterate through remote connections, the remote connection
manager maintains a local cache of connected nodes. Unfortunately this
is difficult in relationship with testing as it is inherently racy in
comparison to the parent connection manager map of connections.

This commit improves the relationship by only returning a cached
connection if it is still registered with the parent. If the connection
is not open, we will go to the slow path of allocating a iterator
directly from the parent.
2020-04-30 12:13:06 -06:00
Igor Motov d8f9df771d
Expose agg usage in Feature Usage API (#55732) (#56048)
Counts usage of the aggs and exposes them on the _nodes/usage/.

Closes #53746
2020-04-30 12:53:36 -04:00
Lee Hinman 3dada1e2d3
[7.x] Handle merging dotted object names when merging V2 template mappings (#55982) (#56041)
Backports the following commits to 7.x:
 - Handle merging dotted object names when merging V2 template mappings (#55982)
2020-04-30 10:51:43 -06:00
Przemko Robakowski 797f63e743
[7.x] Emit deprecation warning if multiple v1 templates match with a new index (#55558) (#56038)
* Emit deprecation warning if multiple v1 templates match with a new index (#55558)

* Emit deprecation warning if multiple v1 templates match with a new index

* DEPRECATION_LOGGER rename
2020-04-30 17:36:17 +02:00
Luca Cavanna fc6422ffcc Consolidate DelayableWriteable (#55932)
This commit includes a number of minor improvements around `DelayableWriteable`: javadocs were expanded and reworded, `get` was renamed to `expand` and `DelayableWriteable` no longer implements `Supplier`. Also a couple of methods are now private instead of package private.
2020-04-30 17:16:58 +02:00
Andrei Dan 68985bc1ca
Add HLRC support for simulate index template api (#55936) (#56029)
(cherry picked from commit 475790c34e0bab95d352132d6be63c4f5b219fb1)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-30 15:40:48 +01:00
Nhat Nguyen 2fd257add2 Ensure no circular reference in translog tragic exception (#55959)
We generate a circular reference exception in translog in 6.8 in the
following scenario:

- The first rollGeneration hits "too many open files" exception when it's
copying a checkpoint file. We will set the tragic exception and close
the translog

- The second rollGeneration hits AlreadyClosedException as the current
writer is closed. We will suppress the ACE to the current tragic
exception. Unfortunately, this leads to a circular reference as ACE
already suppresses the tragic exception.

Other factors that help to manifest this bug:
- We do not fail the engine on AlreadyClosedException in flush
- We do not check for ensureOpen before rolling a new generation

Closes #55893
2020-04-30 08:40:19 -04:00
Christoph Büscher c2afbf20de
Fix ExistsQueryBuilder#testToQuery failure (#56006)
Ocassionally this test can fail when the randomized index.version.created is
before 6.1. In this case we don't check that if mappedFields.size() == 0 we
expect a MatchNoDocsQuery query being returned, which we do for other versions.
This fails only occasionally but with the seed provided on the original issue.
It also shouldn't be an issue on master since we shouldn't test with these pre-7
index versions there.

Closes #55950
2020-04-30 12:28:05 +02:00
Dan Hermann 9bf254fe36
REST test for rolling data streams 2020-04-29 17:34:52 -05:00
Nhat Nguyen b1136948b4 Mute CancellableTasksIT.testDoNotWaitForCompletion 2020-04-29 18:13:03 -04:00
Dan Hermann bf89e485fc
[7.x] Delete index API properly handles backing indices for data streams (#55971) 2020-04-29 16:32:59 -05:00
Christos Soulios 43dab77186
[7.x] Modified searchAndReduce() to return empty agg when no docs exist (#55967)
Backports #55826 to 7.x

    Modified AggregatorTestCase.searchAndReduce() method so that it returns an empty aggregation result when no documents have been inserted.

    Also refactored several aggregation tests so they do not re-implement method AggregatorTestCase.testCase()

    Fixes #55824
2020-04-30 00:28:32 +03:00
Mark Tozzi 9cd3175bbb
[7.x] Wire up AutoDateHistogram to the ValuesSourceRegistry (#55687) (#55870) 2020-04-29 16:26:09 -04:00
Nhat Nguyen c547a92ac6 Revert "Mute CancellableTasksIT.testDoNotWaitForCompletion"
This reverts commit 0c095bbd0c.
2020-04-29 16:21:42 -04:00
Mark Vieira 0c095bbd0c
Mute CancellableTasksIT.testDoNotWaitForCompletion 2020-04-29 12:59:07 -07:00
Nhat Nguyen edbaa19a5d Add trace log for task cancellation (#55940)
Adding trace logs to the task cancellation and its tests 
to debug the test failure in #55875.

Relates ##55875
2020-04-29 15:37:37 -04:00
Mark Vieira 144e8ce092
Update Lucene version for Elasticsearch 6.8.9 (#55963) 2020-04-29 12:36:37 -07:00
Tim Brooks 9eb6736500
Fix NullPointer when message shortcircuited (#55945)
Currently if we shortcircuit a message the breaker release is null since
there is nothing to be broken. However, the TcpTransportChannel
infrastructure still expects it. This commit resolves this issue be
returning a no-op breaker release.
2020-04-29 10:11:39 -06:00
Christoph Büscher 57409fccbd Remove unnecessary instance variable in QueryStringQueryParser (#55915)
Currently `currentFieldType` is an instance variable that is first set and then
used by all methods referring to it. We can make it local to each method
instead, avoiding possible state problems and improve readability of the code
instead.
2020-04-29 16:30:48 +02:00
Andrei Dan 6b886b0b7a
[7.x] Add simulate template composition API _index_template/_simulate_index/{name} (#55686) (#55922)
This adds a new api to simulate matching the given index name against the
 index templates in the system.

The syntax for the new API takes the following form:

POST _index_template/_simulate_index/{index_name}
{
  "index_patterns": ["logs-*"],
  "priority": 15,
  "template": {
	"settings": {
		"number_of_shards": 3
	}
       ...
   }
}

Where the body is optional, but we support the entire body used by the
PUT _index_template/{name} api. When the body is specified we'll simulate
matching the given index against a system that'd have the given index
template together with the index templates that exist in the system.

The response, in both cases, will return the matching template's resolved
settings, mappings and aliases, together with a special field that'll print any
overlapping templates and their corresponding index patterns.

(cherry picked from commit 1a5845edce1f445c58e094e9a3b6792e21e543b0)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-29 14:57:44 +01:00
Christos Soulios 02bf0c586a
[7.x] Histogram field type support for Sum aggregation (#55916)
Implements Sum aggregation over Histogram fields by summing the value of each bucket multiplied by their count as requested in #53285

Backports #55681 to 7.x
2020-04-29 15:06:12 +03:00
Yang Cheng 06b3345787 Avoid double-recovery when state recovery delayed
Today if state recovery is delayed by the `gateway.recover_after_*` settings
then we may end up performing state recovery twice: once when enough nodes have
joined the cluster, and again when the timeout elapses. The second state
recovery reinitializes the routing table, effectively discarding all
recovered/recovering shards and starting again from scratch. This commit adds a
check to prevent this second state recovery.

Closes #55564
2020-04-29 11:55:28 +01:00
Armin Braun b96db2ee2b
Increase Timeout in ClusterDisruptionIT.testRestartNodeWhileIndexing (#55877) (#55880)
The test failed in #55869 but the `docId` was never stuck, it just moved slowly upwards.
=> increasing to timeout.

Closes #55869
2020-04-29 06:47:00 +02:00
Tim Brooks 8d1595698b
Improve start_recovery check in IndexRecoveryIT (#55867)
Currently the testTransientErrorsDuringRecoveryAreRetried validates that
the expected peer recovery starts only once. This check is coarse and is
executed on all nodes and indexes. This commit modifies this check to
only be performed on the expected index. Additionally this commit
removes the disruption behavior from the "blue" node where it is not
relevant. Finally, this commit improves the logging for this test.
2020-04-28 16:40:03 -06:00
Nik Everett a5d0409a8f
Save memory in on aggs in async search (#55683) (#55879)
This replaces a reference to the result of partially reducing
aggregations that async search keeps with a reference to the serialized
form of the result of the partial reduction which we need to keep
anyway.
2020-04-28 16:23:30 -04:00
Ryan Ernst fed296ebb7
Add method to check if object is generically writeable in stream (#54936) (#55561)
When calling scripts in metric aggregation, the returned metric state is
passed along to the coordinating node to do the final reduce. However,
it is possible the object could contain nested state which is unknown to
StreamOutput/StreamInput. This would then result in the node crashing as
exceptions are not expected in the middle of serialization.

This commit adds a method to StreamOutput that can determine if an
object is writeable by the stream. It uses the same logic
writeGenericValue, special casing each of the supported collection types
to recursively determine if each contained value is itself writeable.

relates #54708
2020-04-28 13:08:41 -07:00
Tim Brooks 9e376589a6
Fully stop RetryableAction when cancelled (#55614)
Currently cancelling the RetryableAction does not stop one last run from
being executed. This commit makes a best effort attempt to cancel a
scheduled retry and guards future executions from the action already
being completed.
2020-04-28 13:54:00 -06:00
Tim Brooks cd228095df
Retry failed peer recovery due to transient errors (#55883)
Currently a failed peer recovery action will fail an recovery. This
includes when the recovery fails due to potentially short lived
transient issues such as rejected exceptions or circuit breaking
errors.

This commit adds the concept of a retryable action. A retryable action
will be retryed in face of certain errors. The action will be retried
after an exponentially increasing backoff period. After defined time,
the action will timeout.

This commit only implements retries for responses that indicate the
target node has NOT executed the action.
2020-04-28 13:52:49 -06:00
Nhat Nguyen ad6221c0cb Fix testKeepTranslogAfterGlobalCheckpoint (#55868)
If we advance the global checkpoint during commit and sync that
checkpoint after commit, then the assertions in the test won't hold
because the deletion policy did not see the latest global checkpoint
but only the value before committing.

Closes #55680
2020-04-28 12:50:41 -04:00
Henning Andersen cab7bcc156
Disk decider respect watermarks for single data node (#55805) (#55847)
The disk decider had special handling for the single data node case,
allowing any allocation (skipping watermark checks) for such clusters.
This special handling can now be avoided via a setting.
2020-04-28 18:46:22 +02:00
Lee Hinman 777caf0725
[7.x] Add support for V2 index templates to /_cat/templates (#55829) (#55866)
Backports the following commits to 7.x:
 - Add support for V2 index templates to /_cat/templates (#55829)
2020-04-28 10:14:19 -06:00
Mark Tozzi bebbc375ae
Wire up IpRangeAggregation to ValuesSourceRegistry (#55831) (#55859) 2020-04-28 12:10:21 -04:00
Armin Braun f38385ee25
Fix Leaking Listener When Closing NodeClient (#55676) (#55864)
If a node client (or rather its underlying node) is closed then
any executions on it will just quietly fail as happens in #55660
via closing the nodes on the test thread and asynchronously using
a node client.

Closes #55660
2020-04-28 17:27:58 +02:00
Lee Hinman 3b211c1212
Downgrade template update error to a warning for v1 templates (#55611)
For 7.x, we already implemented the `?prefer_v2_templates` flag and made V2 templates opt-in, so we
can relax the error when updating V1 templates to just a warning. This will still be a hard error
for 8.0+

Relates to #53101
2020-04-28 09:16:08 -06:00
Armin Braun 51a94102e8
Improve some Byte Array Handling Spots (#55844) (#55856)
Some small memory-saving improvements in `byte[]` handling.
2020-04-28 16:38:48 +02:00
Christos Soulios fae9ec13dd
Removed ValuesSourceRegistry.registerAny() (#55846)
* Backports #55747 to 7.x
* All ValuesSourceTypes must be registered
explicitly
* Removed lambdas in ValuesSourceRegistry
2020-04-28 15:44:42 +03:00
Adrien Grand 58c3bb5ae1
Repurpose `ignore_throttled` to be only about frozen indices. (#55047) (#55852)
This has no practical impact on users since frozen indices are the only
throttled indices today. However this has an impact on upcoming features
that would use search throttling.

Filtering out throttled indices made sense a couple years ago, but as
we're now improving support for slow requests with `_async_search` and
exploring ways to reduce storage costs, this feature has most likely
become a trap, that we'd like to not have with upcoming features that
would use search throttling.

Relates #54058
2020-04-28 14:31:54 +02:00
Tim Brooks 80662f31a1
Introduce mechanism to stub request handling (#55832)
Currently there is a clear mechanism to stub sending a request through
the transport. However, this is limited to testing exceptions on the
sender side. This commit reworks our transport related testing
infrastructure to allow stubbing request handling on the receiving side.
2020-04-27 16:57:15 -06:00
Igor Motov 2ff858b290
Fix error massage for unknown value type (#55821) (#55825)
Fixes confusing error message when unknown value type is specified in a terms
aggregation. Adds support for parsing "numeric" and "number" value types.

Fixes #55727
2020-04-27 18:34:43 -04:00
weizijun 08d328333a Append indies to update index setting task name (#55714)
This change adds index names to the name of the update index setting 
task so we have more information about the pending tasks.
2020-04-27 17:50:36 -04:00
Julie Tibshirani 4bfd65a375 Remove TODO around aggregating on _index.
The _index field can in fact be used in aggregations.
2020-04-27 12:48:20 -07:00
Tal Levy 6ba5148ead
Add geo_shape support for the geo_centroid aggregation (#55602) (#55819)
this commit leverages the new geo_shape doc values
to register a new geo_centroid aggregator that works
on geo_shape field.
2020-04-27 12:16:10 -07:00