Commit Graph

4661 Commits

Author SHA1 Message Date
Julie Tibshirani e852bb29b7
Simplify signature of FieldMapper#parseCreateField. (#56144)
`FieldMapper#parseCreateField` accepts the parse context, plus a list of fields
as an output parameter. These fields are immediately added to the document
through `ParseContext#doc()`.

This commit simplifies the signature by removing the list of fields, and having
the mappers add the fields directly to `ParseContext#doc()`. I think this is
nicer for implementors, because previously fields could be added either through
the list, or the context (through `add`, `addWithKey`, etc.)
2020-05-06 11:12:09 -07:00
Rory Hunter 350e930e55 Revert "Add version 7.9.0"
This reverts commit b8b4ebd089.
2020-05-06 14:24:55 +01:00
Rory Hunter b8b4ebd089 Add version 7.9.0 2020-05-06 09:12:14 +01:00
Tanguy Leroux 131a3911eb Replace BlobContainerWrapper by FilterBlobContainer (#56200)
A FilterBlobContainer class was introduced in #55952 and it delegates
 its behavior to a given BlobContainer while allowing to override 
only necessary methods.

This commit replaces the existing BlobContainerWrapper class from 
the test framework with the new FilterBlobContainer from core.
2020-05-06 10:05:43 +02:00
Dan Hermann 6674f14fb3
[7.x] Get index includes parent data stream for backing indices (#56238) 2020-05-05 15:43:42 -05:00
Mark Tozzi 33da086d7b
[7.x] Wire up DiversifiedAggregation (#56145) (#56222) 2020-05-05 13:11:49 -04:00
Tal Levy e4f2c3105d
Add geo_shape support for geotile_grid and geohash_grid (#55966) (#56228)
this commit adds aggregation support for the geo_shape field
type on geo*_grid aggregations.

it introduces a Tiler for both tiles and hashes that enables a new type of
ValuesSource to replace the GeoPoint's CellIdSource. This makes it possible
for the existing Aggregator to be re-used, so no new implementations of
the grid aggregators are added.
2020-05-05 09:54:14 -07:00
Lee Hinman b77c0bbe26
[7.x] Validate V2 templates more strictly (#56170) (#56226)
Backports the following commits to 7.x:
 - Validate V2 templates more strictly (#56170)
2020-05-05 10:34:56 -06:00
Igor Motov 94b349cd18
[7.x] Simplify the ValuesSourceRegistry structure (#56154) (#56197)
Follow up to #55747.
2020-05-05 10:37:02 -04:00
Tanguy Leroux b9636713b1
Searchable Snapshots should respect max_restore_bytes_per_sec (#55952) (#56199)
This commit changes searchable snapshots so that it now respects the 
repository's max_restore_bytes_per_sec setting when it downloads blobs.

Backport of #55952 for 7.x
2020-05-05 15:43:06 +02:00
Jason Tedor c38388c506
Fix compiling in TransportValidateQueryActionTests
This arose after a backport where we do not have the nicities of the
Java 11 diamond operator. This commit fixes it by adding the proper type
parameter.
2020-05-05 07:36:40 -04:00
Jason Tedor 410eb29937
Fix validate query listener invocation bug (#56157)
When the index we are validating a query does not exist, we try to send
back a response letting the client know that the index does not
exist. Yet, we accidentally fallthrough into the case that the
validation failed for some other reason. This means that we end up
notifying the channel twice. Sometimes the notification occurs after the
failure has been written out and the channel closed (so the second
invocation leads to a silent failed to write to a closed channel issue),
and sometimes the response does end up in the channel, creating garbled
responses to the client. This commit fixes that issue by avoiding the
fallthrough.
2020-05-05 07:26:02 -04:00
Nhat Nguyen 60d097e262
Avoid copying file chunks in peer covery (#56072) (#56172)
A follow-up of #55353 to avoid copying file chunks before sending
them to the network layer.

Relates #55353
2020-05-04 23:39:34 -04:00
Ryan Ernst 39ba06cbb2
Add dummy file for new client example snippets location (#56152)
This file is added simply to ensure the new directory exists, so it can
be added to the docs configuration.
2020-05-04 15:48:56 -07:00
Lee Hinman 8fa14b333d
[7.x] Validate non-negative priorities for V2 index templates (#56139) (#56163)
Backports the following commits to 7.x:
 - Validate non-negative priorities for V2 index templates (#56139)
2020-05-04 16:19:13 -06:00
Martijn van Groningen 2ac32db607
Move includeDataStream flag from IndicesOptions to IndexNameExpressionResolver.Context (#56151)
Backport of #56034.

Move includeDataStream flag from an IndicesOptions to IndexNameExpressionResolver.Context
as a dedicated field that callers to IndexNameExpressionResolver can set.

Also alter indices stats api to support data streams.
The rollover api uses this api and otherwise rolling over data stream does no longer work.

Relates to #53100
2020-05-04 22:38:33 +02:00
Lee Hinman 3cefe192a2
[7.x] Remove Index Templates V2 feature flag (#56123) (#56141)
Backports the following commits to 7.x:
 - Remove Index Templates V2 feature flag (#56123)
2020-05-04 13:15:51 -06:00
Martijn van Groningen 6d03081560
Add auto create action (#56122)
Backport of #55858 to 7.x branch.

Currently the TransportBulkAction detects whether an index is missing and
then decides whether it should be auto created. The coordination of the
index creation also happens in the TransportBulkAction on the coordinating node.

This change adds a new transport action that the TransportBulkAction delegates to
if missing indices need to be created. The reasons for this change:

* Auto creation of data streams can't occur on the coordinating node.
Based on the index template (v2) either a regular index or a data stream should be created.
However if the coordinating node is slow in processing cluster state updates then it may be
unaware of the existence of certain index templates, which then can load to the
TransportBulkAction creating an index instead of a data stream. Therefor the coordination of
creating an index or data stream should occur on the master node. See #55377

* From a security perspective it is useful to know whether index creation originates from the
create index api or from auto creating a new index via the bulk or index api. For example
a user would be allowed to auto create an index, but not to use the create index api. The
auto create action will allow security to distinguish these two different patterns of
index creation.
This change adds the following new transport actions:

AutoCreateAction, the TransportBulkAction redirects to this action and this action will actually create the index (instead of the TransportCreateIndexAction). Later via #55377, can improve the AutoCreateAction to also determine whether an index or data stream should be created.

The create_index index privilege is also modified, so that if this permission is granted then a user is also allowed to auto create indices. This change does not yet add an auto_create index privilege. A future change can introduce this new index privilege or modify an existing index / write index privilege.

Relates to #53100
2020-05-04 19:10:09 +02:00
Julie Tibshirani 6b5cf1b031 For constant_keyword, make sure exists query handles missing values. (#55757)
It's possible for a constant_keyword to have a 'null' value before any documents
are seen that contain a value for the field. In this case, no documents have a
value for the field, and 'exists' queries should return no documents.
2020-05-04 09:41:52 -07:00
Armin Braun e8ef44ce78
Allow Bulk Snapshot Deletes to Abort (#56009) (#56111)
Making use of #55773 to simplify snapshot state machine.
1. Deletes with no in-progress snapshot now add the delete entry to the cluster state right away
instead of doing a second CS update after the fist update was a NOOP.
2. If a bulk delete matches in-progress as well as completed snapshots, abort the in-progress snapshot
and then move on to delete from the repository.
2020-05-04 16:21:00 +02:00
Christos Soulios c65f828cb7
[7.x] Histogram field type support for ValueCount and Avg aggregations (#56099)
Backports #55933 to 7.x

Implements value_count and avg aggregations over Histogram fields as discussed in #53285

- value_count returns the sum of all counts array of the histograms
- avg computes a weighted average of the values array of the histogram by multiplying each value with its associated element in the counts array
2020-05-04 13:23:02 +03:00
Armin Braun e01b999ef0
Add Functionality to Consistently Read RepositoryData For CS Updates (#55773) (#56091)
Using optimistic locking, add the ability to run a repository state
update task with a consistent view of the current repository data.
Allows for a follow-up to remove the snapshot INIT state.
2020-05-04 08:13:14 +02:00
Armin Braun 3a64ecb6bf
Allow Deleting Multiple Snapshots at Once (#55474) (#56083)
* Allow Deleting Multiple Snapshots at Once (#55474)

Adds deleting multiple snapshots in one go without significantly changing the mechanics of snapshot deletes otherwise.
This change does not yet allow mixing snapshot delete and abort. Abort is still only allowed for a single snapshot delete by exact name.
2020-05-03 20:30:58 +02:00
David Turner 69f50fe79f Improve same-shard allocation explanations (#56010)
I see occasional confusion about the explanations emitted by the same-shard
allocation decider, particularly amongst new users setting up a single-node
cluster and trying to determine why their cluster has `yellow` health. For
example:

    the shard cannot be allocated to the same node on which a copy of the shard
    already exists

This is technically correct but it's quite a complicated sentence. Also, by
starting with "the shard cannot be allocated" it makes it sound like this is
the problem, whereas in fact this message is a good thing and users should
typically focus their attention elsewhere.

This commit simplifies the wording of these messages and makes them sound more
positive, for example:

    a copy of this shard is already allocated to this node
2020-05-01 10:07:14 +01:00
Mark Tozzi d8eb51ed63
Wire up GeoDistanceAggregation (#55975) (#56042) 2020-04-30 15:43:27 -04:00
Tim Brooks 54dbea6c65
Improve RemoteConnectionManager consistency (#55759)
In order to iterate through remote connections, the remote connection
manager maintains a local cache of connected nodes. Unfortunately this
is difficult in relationship with testing as it is inherently racy in
comparison to the parent connection manager map of connections.

This commit improves the relationship by only returning a cached
connection if it is still registered with the parent. If the connection
is not open, we will go to the slow path of allocating a iterator
directly from the parent.
2020-04-30 12:13:06 -06:00
Igor Motov d8f9df771d
Expose agg usage in Feature Usage API (#55732) (#56048)
Counts usage of the aggs and exposes them on the _nodes/usage/.

Closes #53746
2020-04-30 12:53:36 -04:00
Lee Hinman 3dada1e2d3
[7.x] Handle merging dotted object names when merging V2 template mappings (#55982) (#56041)
Backports the following commits to 7.x:
 - Handle merging dotted object names when merging V2 template mappings (#55982)
2020-04-30 10:51:43 -06:00
Przemko Robakowski 797f63e743
[7.x] Emit deprecation warning if multiple v1 templates match with a new index (#55558) (#56038)
* Emit deprecation warning if multiple v1 templates match with a new index (#55558)

* Emit deprecation warning if multiple v1 templates match with a new index

* DEPRECATION_LOGGER rename
2020-04-30 17:36:17 +02:00
Luca Cavanna fc6422ffcc Consolidate DelayableWriteable (#55932)
This commit includes a number of minor improvements around `DelayableWriteable`: javadocs were expanded and reworded, `get` was renamed to `expand` and `DelayableWriteable` no longer implements `Supplier`. Also a couple of methods are now private instead of package private.
2020-04-30 17:16:58 +02:00
Andrei Dan 68985bc1ca
Add HLRC support for simulate index template api (#55936) (#56029)
(cherry picked from commit 475790c34e0bab95d352132d6be63c4f5b219fb1)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-30 15:40:48 +01:00
Nhat Nguyen 2fd257add2 Ensure no circular reference in translog tragic exception (#55959)
We generate a circular reference exception in translog in 6.8 in the
following scenario:

- The first rollGeneration hits "too many open files" exception when it's
copying a checkpoint file. We will set the tragic exception and close
the translog

- The second rollGeneration hits AlreadyClosedException as the current
writer is closed. We will suppress the ACE to the current tragic
exception. Unfortunately, this leads to a circular reference as ACE
already suppresses the tragic exception.

Other factors that help to manifest this bug:
- We do not fail the engine on AlreadyClosedException in flush
- We do not check for ensureOpen before rolling a new generation

Closes #55893
2020-04-30 08:40:19 -04:00
Christoph Büscher c2afbf20de
Fix ExistsQueryBuilder#testToQuery failure (#56006)
Ocassionally this test can fail when the randomized index.version.created is
before 6.1. In this case we don't check that if mappedFields.size() == 0 we
expect a MatchNoDocsQuery query being returned, which we do for other versions.
This fails only occasionally but with the seed provided on the original issue.
It also shouldn't be an issue on master since we shouldn't test with these pre-7
index versions there.

Closes #55950
2020-04-30 12:28:05 +02:00
Dan Hermann 9bf254fe36
REST test for rolling data streams 2020-04-29 17:34:52 -05:00
Nhat Nguyen b1136948b4 Mute CancellableTasksIT.testDoNotWaitForCompletion 2020-04-29 18:13:03 -04:00
Dan Hermann bf89e485fc
[7.x] Delete index API properly handles backing indices for data streams (#55971) 2020-04-29 16:32:59 -05:00
Christos Soulios 43dab77186
[7.x] Modified searchAndReduce() to return empty agg when no docs exist (#55967)
Backports #55826 to 7.x

    Modified AggregatorTestCase.searchAndReduce() method so that it returns an empty aggregation result when no documents have been inserted.

    Also refactored several aggregation tests so they do not re-implement method AggregatorTestCase.testCase()

    Fixes #55824
2020-04-30 00:28:32 +03:00
Mark Tozzi 9cd3175bbb
[7.x] Wire up AutoDateHistogram to the ValuesSourceRegistry (#55687) (#55870) 2020-04-29 16:26:09 -04:00
Nhat Nguyen c547a92ac6 Revert "Mute CancellableTasksIT.testDoNotWaitForCompletion"
This reverts commit 0c095bbd0c.
2020-04-29 16:21:42 -04:00
Mark Vieira 0c095bbd0c
Mute CancellableTasksIT.testDoNotWaitForCompletion 2020-04-29 12:59:07 -07:00
Nhat Nguyen edbaa19a5d Add trace log for task cancellation (#55940)
Adding trace logs to the task cancellation and its tests 
to debug the test failure in #55875.

Relates ##55875
2020-04-29 15:37:37 -04:00
Mark Vieira 144e8ce092
Update Lucene version for Elasticsearch 6.8.9 (#55963) 2020-04-29 12:36:37 -07:00
Tim Brooks 9eb6736500
Fix NullPointer when message shortcircuited (#55945)
Currently if we shortcircuit a message the breaker release is null since
there is nothing to be broken. However, the TcpTransportChannel
infrastructure still expects it. This commit resolves this issue be
returning a no-op breaker release.
2020-04-29 10:11:39 -06:00
Christoph Büscher 57409fccbd Remove unnecessary instance variable in QueryStringQueryParser (#55915)
Currently `currentFieldType` is an instance variable that is first set and then
used by all methods referring to it. We can make it local to each method
instead, avoiding possible state problems and improve readability of the code
instead.
2020-04-29 16:30:48 +02:00
Andrei Dan 6b886b0b7a
[7.x] Add simulate template composition API _index_template/_simulate_index/{name} (#55686) (#55922)
This adds a new api to simulate matching the given index name against the
 index templates in the system.

The syntax for the new API takes the following form:

POST _index_template/_simulate_index/{index_name}
{
  "index_patterns": ["logs-*"],
  "priority": 15,
  "template": {
	"settings": {
		"number_of_shards": 3
	}
       ...
   }
}

Where the body is optional, but we support the entire body used by the
PUT _index_template/{name} api. When the body is specified we'll simulate
matching the given index against a system that'd have the given index
template together with the index templates that exist in the system.

The response, in both cases, will return the matching template's resolved
settings, mappings and aliases, together with a special field that'll print any
overlapping templates and their corresponding index patterns.

(cherry picked from commit 1a5845edce1f445c58e094e9a3b6792e21e543b0)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-29 14:57:44 +01:00
Christos Soulios 02bf0c586a
[7.x] Histogram field type support for Sum aggregation (#55916)
Implements Sum aggregation over Histogram fields by summing the value of each bucket multiplied by their count as requested in #53285

Backports #55681 to 7.x
2020-04-29 15:06:12 +03:00
Yang Cheng 06b3345787 Avoid double-recovery when state recovery delayed
Today if state recovery is delayed by the `gateway.recover_after_*` settings
then we may end up performing state recovery twice: once when enough nodes have
joined the cluster, and again when the timeout elapses. The second state
recovery reinitializes the routing table, effectively discarding all
recovered/recovering shards and starting again from scratch. This commit adds a
check to prevent this second state recovery.

Closes #55564
2020-04-29 11:55:28 +01:00
Armin Braun b96db2ee2b
Increase Timeout in ClusterDisruptionIT.testRestartNodeWhileIndexing (#55877) (#55880)
The test failed in #55869 but the `docId` was never stuck, it just moved slowly upwards.
=> increasing to timeout.

Closes #55869
2020-04-29 06:47:00 +02:00
Tim Brooks 8d1595698b
Improve start_recovery check in IndexRecoveryIT (#55867)
Currently the testTransientErrorsDuringRecoveryAreRetried validates that
the expected peer recovery starts only once. This check is coarse and is
executed on all nodes and indexes. This commit modifies this check to
only be performed on the expected index. Additionally this commit
removes the disruption behavior from the "blue" node where it is not
relevant. Finally, this commit improves the logging for this test.
2020-04-28 16:40:03 -06:00
Nik Everett a5d0409a8f
Save memory in on aggs in async search (#55683) (#55879)
This replaces a reference to the result of partially reducing
aggregations that async search keeps with a reference to the serialized
form of the result of the partial reduction which we need to keep
anyway.
2020-04-28 16:23:30 -04:00