Today we try to pull stats from index writer but we do not get a
consistent view of stats. Under heavy indexing, this inconsistency can
be very skewed indeed. In particular, it can lead to the number of
deleted docs being reported as negative and this leads to serialization
issues. Instead, we should provide a consistent view of the stats by
using an index reader.
Relates #22317
The backwards compatibility tests rely on gradle's built-in mechanisms for resolving dependencies
to get the zip of the older version we test against. By default, this will cache snapshots for
24 hours, which can lead to unexpected failures in CI. This change makes the special configurations
for backwards compatibility always update their snapshots by setting the amount of time to cache
to 0 seconds.
Not doing this made it difficult to establish a happens before relationship between connecting to a node and adding a listeners. Causing test code like this to fail sproadically:
```
// connection to reuse
handleA.transportService.connectToNode(handleB.node);
// install a listener to check that no new connections are made
handleA.transportService.addConnectionListener(new TransportConnectionListener() {
@Override
public void onConnectionOpened(DiscoveryNode node) {
fail("should not open any connections. got [" + node + "]");
}
});
```
relates to #22277
Today we silently ignore invalid test logging annotations. This commit
rejects these annotations, failing the processing of the annotation and
aborting the test.
This commit adds a test for applying logging levels in hierarchical
order, and addresses an issue with restoring the logging levels at the
end of a test or suite.
This commit factors out the cluster state update tasks that are published (ClusterStateUpdateTask) from those that are not (LocalClusterUpdateTask), serving as a basis for future refactorings to separate the publishing mechanism out of ClusterService.
We have to sort the logger names so they wouldn't override each other. Processing org.elasticsearch:DEBUG after org.elasticsearch.transport:TRACE resets the setting of the later
When starting a standalone cluster, we do not able assertions. This is
problematic because it means that we miss opportunities to catch
bugs. This commit enables assertions for standalone integration tests,
and fixes a couple bugs that were uncovered by enabling these.
Relates #22334
We don't *want* to use negative numbers with `writeVLong`
so throw an exception when we try. On the other
hand unforeseen bugs might cause us to write negative numbers (some versions of Elasticsearch don't have the exception, only an assertion)
so this fixes `readVLong` so that instead of reading a wrong
value and corrupting the stream it reads the negative value.
Optimistically check for `tag` of an unknown processor for better tracking of which
processor declaration is to blame in an invalid configuration.
Closes#21429.
* Remove a checked exception, replacing it with `ParsingException`.
* Remove all Parser classes for the yaml sections, replacing them with static methods.
* Remove `ClientYamlTestFragmentParser`. Isn't used any more.
* Remove `ClientYamlTestSuiteParseContext`, replacing it with some static utility methods.
I did not rewrite the parsers using `ObjectParser` because I don't think it is worth it right now.
Currently we only apply date detection on strings that contain either `:`, `-`
or `/`. This commit inverses the heuristic in order to only apply date detection
on strings that are not parseable as a number, so that more date formats can be
used as dynamic dates formats.
Closes#1694
Today we only expose `value_type` in scriptable aggregations, however it is
also useful with unmapped fields. I suspect we never noticed because
`value_type` was not documented (fixed) and most aggregations are scriptable.
Closes#20163
`ShardCoreKeyMap.add` is called on each segment for all search requests, which
means it might become a bottleneck under a cocurrent load of cheap search
requests since this method acquires a mutex. This change proposes to use a
`ConcurrentHashMap` which allows to only take the mutex in the case that the
`LeafReader` has never been seen before.
This adds test classes that can be used to test the wire serialisation and (optionally) the XContent serialisation of objects that implement Streamable/Writeable and ToXContent.
These test classes will enable classes sich as InternalAggregation (or at least its implementations) to be tested in a consistent way when is comes to testing serialisation.
As the translog evolves towards a full operations log as part of the
sequence numbers push, there is a need for the translog to be able to
represent operations for which a sequence number was assigned, but the
operation did not mutate the index. Examples of how this can arise are
operations that fail after the sequence number is assigned, and gaps in
this history that arise when an operation is assigned a sequence number
but the operation never completed (e.g., a node crash). It is important
that these operations appear in the history so that they can be
replicated and replayed during recovery as otherwise the history will be
incomplete and local checkpoints will not be able to advance. This
commit introduces a no-op to the translog to set the stage for these
efforts.
Relates #22291
Today if an older version of a plugin exists, we fail to notify the user
with a helpful error message. This happens because during plugin
verification, we attempt to read the plugin descriptors for all existing
plugins. When an older version of a plugin is sitting on disk, we will
attempt to read this old plugin descriptor and fail due to a version
mismatch. This leads to an unhelpful error message. Instead, we should
check for existence of the plugin as part of the verification phase, but
before attempting to read plugin descriptors for existing plugins. This
enables us to provide a helpful error message to the user.
Relates #22305
* Repeated language analyzers
The `catalan` analyzer was repeated on the supported list :)
* Reordered the languages to have alphabetic order
* Added space for format
* Reordered the languages and removed repeated
Added a new section detailing how to use the attachment processor
within an array.
This reverts commit #22296 and instead links to the foreach processor.
The deprecation warning gives now the same message as 5.x. The deprecation warning was previously removed, but given that we are still lenient with old indices we should still output the warning.
Our `float`/`double` fields generally assume that `-0` compares less than `+0`,
except when bounds are exclusive: an exclusive lower bound on `-0` excludes
`+0` and an exclusive upper bound on `+0` excludes `-0`.
Closes#22167
Today we ship with default jvm.options for server Elasticsearch that
prevents Netty from using some unsafe optimizations. Yet, the settings
do nothing for the transport client since it is embedded in other
applications that will not read and use those settings. This commit adds
these settings for the transport client, and is done so in a way that
still enables users to go unsafe if they want to go unsafe (they
shouldn't, but the option is there).
Relates #22284
The way aggregations on scripts work is by hiding scripts behind the same API
that we use for regular fields. However, there is no native support for boolean
fields, those need to be exposed as integers, with `0` standing for `false` and
`1` for true.
Relates #20941
The `UnicastZenPing` shows it's age and is the result of many small changes. The current state of affairs is confusing and is hard to reason about. This PR cleans it up (while following the same original intentions). Highlights of the changes are:
1) Clear 3 round flow - no interleaving of scheduling.
2) The previous implementation did a best effort attempt to wait for ongoing pings to be sent and completed. The pings were guaranteed to complete because each used the total ping duration as a timeout. This did make it hard to reason about the total ping duration and the flow of the code. All of this is removed now and ping should just complete within the given duration or not be counted (note that it was very handy for testing, but I move the needed sync logic to the test).
3) Because of (2) the pinging scheduling changed a bit, to give a chance for the last round to complete. We now ping at the beginning, 1/3 and 2/3 of the duration.
4) To offset for (3) a bit, incoming ping requests are now added to on going ping collections.
5) UnicastZenPing never establishes full blown connections (but does reuse them if there). Relates to #22120
6) Discovery host providers are only used once per pinging round. Closes#21739
7) Usage of the ability to open a connection without connecting to a node ( #22194 ) and shorter connection timeouts helps with connections piling up. Closes#19370
8) Beefed up testing and sped them up.
9) removed light profile from production code
Jason Tedor
Jay Modi
Boaz Leskes
Yannick Welsch
David Pilato
Adrien Grand
gameldar
Tim B
Nik Everett
Tal Levy
Stéphane Campinas
Martijn van Groningen
Colin Goodheart-Smithe
Gameldar
Francesc Gil
Luca Cavanna
Nic Palmer
Rui Hu
javanna