Today when canceling a task we broadcast ban/unban requests to all nodes
in the cluster. This strategy does not scale well for hierarchical
cancellation. With this change, we will track outstanding child requests
and broadcast the cancellation to only nodes that have outstanding child
tasks. This change also prevents a parent task from sending child
requests once it got canceled.
Relates #50990
Supersedes #51157
Co-authored-by: Igor Motov <igor@motovs.org>
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
This commit addresses a long-standing `// TODO` in the coordinator tests to
ensure that the correct no-master block is applied when a node restarts while
disconnected from the cluster.
It also strengthens this test to check that the no-master block is applied
correctly on all nodes, not just the previous master.
Test `testAckListenerReceivesNacksFromFollowerInHigherTerm` was suppressed as
when it was written it didn't work due the lack of proper term bumping. We
added term bumping but never got around to implementing this test. This commit
addresses this.
Removing a few spots where we clearly don't have to fork to the generic or management
pool since either we only interpret the current cluster state or fork-off directly to
some other pool in the transport action logic anyway.
We should never write a circular reference exception as we will fail a
node with StackOverflowError. However, we have one in #53589.
I tried but failed to find its location. With this commit, we will avoid
StackOverflowError in production and detect circular exceptions in
tests.
Closes#53589
We recently cleaned up the use of the word "metadata" across the
codebase. A few additional uses have trickled in, likely from
in-progress work. This commit cleans up these last few instances.
Relates #54519
* Use V2 index templates during index creation
This commit changes our index creation code to use (and favor!) V2 index templates during index
creation. The creation precedence goes like so, in order of precedence:
- Existing source `IndexMetadata` - for example, when recovering from a peer or a shrink/split/clone
where index templates should not be applied
- A matching V2 index template, if one is found
- When a V2 template is found, all component templates (in the `composed_of` field) are applied
in the order that they appear, with the index template having the 2nd highest precedence (the
create index request always has the top priority when it comes to index settings)
- All matching V1 templates (the old style)
This also adds index template validation when `PUT`-ing a new v2 index template (because this was
required) and ensures that all index and component templates specify *no* top-level mapping type (it
is automatically added when the template is added to the cluster state).
This does not yet implement fine-grained component template merging of mappings, where we favor
merging only a single field's configuration, that will be done in subsequent work.
This also keeps the existing hidden index behavior present for v1 templates, where a hidden index
will match v2 index templates unless they are global (`*`) templates.
Relates to #53101
Some field name constants were not updaten when we moved from "string" to "text"
and "keyword" fields. Renaming them makes it easier and faster to know which
field type is used in test subclassing this base test case.
The test had errors around time units that have different length - think
leap years or months that aren't 30 days. This fixes those errors. In
the proces I've changed a bunch of things to debug the problem:
* Replace `currentTimeMillis` with a random time. Now the test fails
randomly! Wonderful. Much better than on random days of the month.
* Generate buckets "closer together" to test random reduction. Without
this we were super frequently getting stuck in the "year of century"
rounding because *some* of the of the buckets we built were far apart.
This generates a much greater variety of tests.
* Implement `toString` on `RoundingInfo` so I can debug without going
crazy.
* Switch keys in the bucket assertions from epoch millis to `Instant`s
so we can read the failures.
Closes#54540Closes#39497
from `*_flag_registered` to `#_feature_enabled`.
This previous name indicated that a flag was registered,
whilst the feature flag actually controls whether a
feature is enabled.
This commit workarounds a bug in the JDK 14 compiler. It is choking on a
method reference, so we substitute a lambda expression instead. The JDK
bug ID is 9064309.
In #33933 we disallowed changing the `enabled` parameter in object mappings.
However, the fix didn't cover the root object mapper. This PR adjusts the change
to also include the root mapper and clarifies the error message.
Removes pipeline aggregations from the aggregation result tree as they
are no longer used. This stops us from building the pipeline aggregators
at all on data nodes except for backwards compatibility serialization.
This will save a tiny bit of space in the aggregation tree which is
lovely, but the biggest benefit is that it is a step towards simplifying
pipeline aggregators.
This only does about half of the work to remove the pipeline aggs from
the tree. Removing all of it would, well, double the size of the change
and make it harder to review.
I derped out on a last minute bug fix when backporting #54282 and it
only causes the tests to fail about half the time. So I didn't catch
it until after merging. Great! This fixes it.
- Consolidates HDR/TDigest factories into a single factory
- Consolidates most HDR/TDigest builder into an abstract builder
- Deprecates method(), compression(), numSigFig() in favor of a new
unified PercentileConfig object
- Disallows setting algo options that don't apply to current algo
The unified config method carries both the method and algo-specific
setting. This provides a mechanism to reject settings that apply
to the wrong algorithm. For BWC the old methods are retained
but marked as deprecated, and can be removed in future versions.
Co-authored-by: Mark Tozzi <mark.tozzi@gmail.com>
Co-authored-by: Mark Tozzi <mark.tozzi@gmail.com>
This fixes pipeline aggregations used in cross cluster search from an older
version of Elasticsearch to a newer version of Elasticsearch. I broke
this in #53730 when I was too aggressive in shutting off serialization
of pipeline aggs. In particular, this comes up when the coordinating
node is pre-7.8.0 and the gateway node is on or after 7.8.0.
The fix is another step down the line to remove pipeline aggregators
from the aggregation tree. Sort of. It create a new
`List<PipelineAggregator>` member in `InternalAggregation` *but* it is
only used for bwc serialization and it is fed by the mechanism
established in #53730 to read the pipelines from the
Right now you can't tell from the task description whether or not the
search is a scroll. This adds that information to the description which
is super useful if you are trying to debug a cluster that is running out
of scroll contexts.
Today the usage service can let in some issues, such as handlers that do
not have a name, where the errors do not manifest until later (calling
the usage API), or conflicting handlers with the same name. This commit
addresses this by adding some validation to the usage service.
Adds tests for supported ValuesSourceTypes, unmapped fields, scripting,
and the missing param. The tests for unmapped fields and scripting are
migrated from the StatsIT integration test
This adds tests for supported ValuesSourceTypes, unmapped fields,
scripting, and the missing param. The tests for unmapped fields and
scripting are migrated from the SumIT integration test
Tests for unmapped fields, the missing parameter, scripting, and correct
ValuesSource types in MissingAggregatorTests. Basic yaml tests for the
missing agg
For #42949
* Refactor nodes stats request builders to match requests (#54363)
* Remove hard-coded setters from NodesInfoRequestBuilder
* Remove hard-coded setters from NodesStatsRequest
* Use static imports to reduce clutter
* Remove uses of old info APIs
Because -1 is technically a valid TimeValue (as a sentinel value), that is now
explicitly checked for when validating gc_thresholds. The tests are also
adjusted to test this case separately from other negative values.
Refactor SearchHit to have separate document and meta fields.
This is a part of bigger refactoring of issue #24422 to remove
dependency on MapperService to check if a field is metafield.
Relates to PR: #38373
Relates to issue #24422
Co-authored-by: sandmannn <bohdanpukalskyi@gmail.com>
This is a follow up to a previous commit that renamed MetaData to
Metadata in all of the places. In that commit in master, we renamed
META_DATA to METADATA, but lost this on the backport. This commit
addresses that.
This is a simple naming change PR, to fix the fact that "metadata" is a
single English word, and for too long we have not followed general
naming conventions for it. We are also not consistent about it, for
example, METADATA instead of META_DATA if we were trying to be
consistent with MetaData (although METADATA is correct when considered
in the context of "metadata"). This was a simple find and replace across
the code base, only taking a few minutes to fix this naming issue
forever.
* Comprehensively test supported/unsupported field type:agg combinations (#52493)
This adds a test to AggregatorTestCase that allows us to programmatically
verify that an aggregator supports or does not support a particular
field type. It fetches the list of registered field type parsers,
creates a MappedFieldType from the parser and then attempts to run
a basic agg against the field.
A supplied list of supported VSTypes are then compared against the
output (success or exception) and suceeds or fails the test accordingly.
Co-Authored-By: Mark Tozzi <mark.tozzi@gmail.com>
* Skip fields that are not aggregatable
* Use newIndexSearcher() to avoid incompatible readers (#52723)
Lucene's `newSearcher()` can generate readers like ParallelCompositeReader
which we can't use. We need to instead use our helper `newIndexSearcher`
The docs here add nothing compared to those in the package. If anything
they are somewhat confusing since they don't give all necessary details to understand the snapshot process.
=> remove them and link to the complete docs at the package level
Bulk requests currently keep a reference to all bulk item requests until every one of them has
completed. There is no need to do so, however, and, in case of large bulks, can mean
unnecessary holding onto memory that might be better used elsewhere. More so as different
shard-level bulks can complete at different speeds, and one slow shard-level request should
not require holding onto every other shard-level request.
Currently there is an issue with the InboundPipeline releasing bytes
earlier than appropriate. This can lead to the bytes being reused before
the message is handled. This commit fixes that issue and adds a test to
detect when it is occurring.
* Add warnings/errors when V2 templates would match same indices… (#54367)
* Add warnings/errors when V2 templates would match same indices as V1
With the introduction of V2 index templates, we want to warn users that templates they put in place
might not take precedence (because v2 templates are going to "win"). This adds this validation at
`PUT` time for both V1 and V2 templates with the following rules:
** When creating or updating a V2 template
- If the v2 template would match indices for an existing v1 template or templates, provide a
warning (through the deprecation logging so it shows up to the client) as well as logging the
warning
The v2 warning looks like:
```
index template [my-v2-template] has index patterns [foo-*] matching patterns from existing older
templates [old-v1-template,match-all-template] with patterns (old-v1-template =>
[foo*],match-all-template => [*]); this template [my-v2-template] will take
precedence during new index creation
```
** When creating a V1 template
- If the v1 template is for index patterns of `"*"` and a v2 template exists, warn that the v2
template may take precedence
- If the v1 template is for index patterns other than all indices, and a v2 template exists that
would match, throw an error preventing creation of the v1 template
** When updating a V1 template (without changing its existing `index_patterns`!)
- If the v1 template is for index patterns that would match an existing v2 template, warn that the
v2 template may take precedence.
The v1 warning looks like:
```
template [my-v1-template] has index patterns [*] matching patterns from existing index templates
[existing-v2-template] with patterns (existing-v2-template => [foo*]); this template [my-v1-template] may be ignored in favor of an index template at index creation time
```
And the v1 error looks like:
```
template [my-v1-template] has index patterns [foo*] matching patterns from existing index templates
[existing-v2-template] with patterns (existing-v2-template => [f*]), use index templates (/_index_template) instead
```
Relates to #53101
* Remove v2 index and component templates when cleaning up tests
* Finish half-finished comment sentence
* Guard template removal and ignore for earlier versions of ES
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Also ignore 500 errors when clearing index template v2 templates
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This fixes a serialization bug in `auto_date_histogram` that comes up in
a cluster mixed between pre-7.3.0 and post-7.3.0.
Includes #54429 to keep 7.x looking like master for simpler backports.
Closes#54382
Pipeline aggregations like `stats_bucket`, `sum_bucket`, and
`percentiles_bucket` only operate on buckets that have multiple buckets.
This adds support for those aggregations to `geo_distance`, `ip_range`,
`auto_date_histogram`, and `rare_terms`.
This all happened because we used a marker interface to mark compatible
aggs, `MultiBucketAggregationBuilder` and it was fairly easy to forget
to implement the interface.
This replaces the marker interface with an abstract method in
`AggregationBuilder`, `bucketCardinality` which makes you return `NONE`,
`ONE`, or `MANY`. The `bucket` aggregations can check for `MANY`. At
this point `ONE` and `NONE` amount to about the same thing, but I
suspect that'll be a useful distinction when validating bucket sorts.
Closes#53215
This commit takes into account the index.number_of_replicas (defaults to
0 - no replicas- ) value when setting an index template. This change
enables the index.wait_for_active_shards value to be interpreted
correctly
(cherry picked from commit 07026ac3d56dc9fae69467adfda7eaed7ea3ca00)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: tninokehoe <62655306+tninokehoe@users.noreply.github.com>
This commit improves the behavior of aborting snapshots and by that fixes
some extremely rare test failures.
Improvements:
1. When aborting a snapshot while it is in the `INIT` stage we do not need
to ever delete anything from the repository because nothing is written to the
repo during INIT any more (in the past running deletes for these snapshots made
sense because we were writing `snap-` and `meta-` blobs during the `INIT` step).
2. Do not try to finalize snapshots that never moved past `INIT`. Same reason as
with the first step. If we never moved past `INIT` no data was written to the repo
so no need to now write a useless entry for the aborted snapshot to `index-N`.
This is especially true, since the reason the snapshot was aborted during `INIT` was
a delete call so the useless empty snapshot just added to `index-N` would be removed
by the subsequent delete that is still waiting anyway.
3. if after aborting a snapshot we wait for it to finish we should not try deleting it
if it failed. If the snapshot failed it means it did not become part of the most recent
`RepositoryData` so a delete for it will needlessly fail with a confusing message about
that snapshot being missing or concurrent repository modification. I moved to throw the snapshot missing exception here because that seems the most user friendly. This allows the user to simply ignore `404` returns from the delete API when using it to make sure a snapshot is aborted+deleted.
Marking this as a non-issue since it doesn't have any negative repercussions other than confusing exceptions on some snapshot aborts.
Closes#52843
Nhat Nguyen
David Turner
Armin Braun
Jason Tedor
Lee Hinman
James Baiera
Dan Hermann
Christoph Büscher
Nik Everett
markharwood
Martijn van Groningen
Julie Tibshirani
Zachary Tong
Andy Bristol
William Brafford
Gordon Brown
Mayya Sharipova
jimczi
Jake Landis
Yannick Welsch
Tim Brooks
Mark Tozzi
Ryan Ernst
Andrei Dan