* As a result of #44665 the collections returned by the deserialization methods on `StreamInput` may be either mutable or immutable now,
this PR adds documentation for that fact
In 7.x we cannot start a new master-eligible node before the cluster has formed
since we first try and update minimum_master_nodes and this is blocked. This
commit changes the test to start a data-only node so that no such adjustment is
necessary.
Relates #44685
Add more logging to indexRandom
Seems that asynchronous indexing from indexRandom sometimes indexes
the same document twice, which will mess up the expected score calculations.
For example, indexing:
{ "index" : {"_id" : "1" } }
{"important" :"phrase match", "less_important": "nothing important"}
{ "index" : {"_id" : "2" } }
{"important" :"nothing important", "less_important" :"phrase match"}
Produces the expected scores: 13.8 for doc1, and 1.38 for doc2
indexing:
{ "index" : {"_id" : "1" } }
{"important" :"phrase match", "less_important": "nothing important"}
{ "index" : {"_id" : "2" } }
{"important" :"nothing important", "less_important" :"phrase match"}
{ "index" : {"_id" : "3" } }
{"important" :"phrase match", "less_important": "nothing important"}
Produces scores: 9.4 for doc1, and 1.96 for doc2 which are found in the
error logs.
Relates to #43144
Fields in JSON logs should be an escaped JSON fields. It is a broken json value at the moment
"stats": "["group1", "group2"]", -> "stats": "[\"group1\", \"group2\"]",
This should later be refactored into a JSON array of strings (the same as types in 7.x)
Today we block access to the pending tasks API before the cluster has recovered
its state. There's no real need to do so, and the master does meaningful work
even before performing state recovery so it might sometimes be useful to allow
access to this API. This commit changes this API to ignore all cluster blocks.
Fixes#44652
Since #44344 we use IndicesOptions.LENIENT_EXPAND_OPEN
when deciding which indices to include in checkpoint
calculation. This change uses the same option when
deciding which indices to search for data and which
indices to get mappings from, otherwise there is a
potential mismatch between the checkpoint details and
what is searched elsewhere.
The field has to be defined in log4j2.properties and should be an
escaped JSON for now (it is a broken JSON at the moment). This should later be refactored into a JSON array
of strings.
Deprecation logger was filtering log entries by key, that means that if two log messages with the same key are logged from different users, then the second log messages will be filtered.
This change allows to log deprecation message with the same key by different users.
relates #41354
backport #44587
* It seems redundant to synchronize here and check that the map hasn't checked via the `isRunning` under the mutex
* The map won't change if under the mutex that locks on all the updates to it
* Without the mutex it's very unlikely to change inside the method call relative to the likelihood of changing until the generic pool where we check for `isRunning` again anyway
-> just remove the synchronization (it's on the IO loop) and check since we do check the running state on the generic pool under the mutex anyway when we actually fail it
We have to copy the field names otherwise we either have a handle of a
list that a caller might mutate or we might mutate when they aren't
expecting it, or worse, a handle of a list that is not mutable (and we
end up mutating the list).
Relates #44665
* Mute failing test
tracked in #44552
* mute EvilSecurityTests
tracking in #44558
* Fix line endings in ESJsonLayoutTests
* Mute failing ForecastIT test on windows
Tracking in #44609
* mute BasicRenormalizationIT.testDefaultRenormalization
tracked in #44613
* fix mute testDefaultRenormalization
* Increase busyWait timeout windows is slow
* Mute failure unconfigured node name
* mute x-pack internal cluster test windows
tracking #44610
* Mute JvmErgonomicsTests on windows
Tracking #44669
* mute SharedClusterSnapshotRestoreIT testParallelRestoreOperationsFromSingleSnapshot
Tracking #44671
* Mute NodeTests on Windows
Tracking #44256
These Azure tests have hard println statements which means we always see
these messages during configuration. Yet, there are unnecessary most of
the time. This commit changes them to use debug logging.
* We only had the `size == 0` optimization in some but not all spots of deserializing collections in this class, fixed the remaining spots.
* Also fixed the a similar spot when deserializing `ThreadContextStruct` that could now be simplified (it was apparently doing it's own version of this optimization for the first map it deserialized before ... but not for the second map -> made it not instantiate anything if both maps are empty since it's always the same object here anyway)
* Optimize some StreamOutput Operations
* Writing numbers byte by byte adds a lot of unnecessary bounds checks to serialization
* Serializing to a threadlocal `byte[]` instead and bulk writing gives about a 50% speedup on `long` and `vlong` (for large numbers) writes and 30% for `int`, `vint` on Linux on an i9
* Using a threadlocal of the maximum string buffer size we used to allocate before also removes allocations when writing strings in general since we now never have to allocate a `byte[]` for that
* And don't have to GC one either resolving the TODO removed here
In https://github.com/elastic/elasticsearch/pull/41913 setting up the
temp dir for ES was moved from the env script to individual cli scripts.
However, moving it to the windows service cli was missed. This commit
restores setting up the temp dir for the windows service control script.
Today when checksumming a plugin zip during plugin install, we read all
of the bytes of the zip into memory at once. When trying to run the
plugin installer on a small heap (say, 64 MiB), this can lead to the
plugin installer running out of memory when checksumming large
plugins. This commit addresses this by reading the plugin bytes in 8 KiB
chunks, thus using a constant amount of memory independent of the size
of the plugin.
This commit converts all remaining TransportRequest and
TransportResponse classes to implement Writeable, and disallows
Streamable implementations.
relates #34389
This commit adds some default CLI JVM options to control the heap size
and the garbage collector used for the CLI tools. We do this because
otherwise the JVM will default to large initial and max heap sizes based
on the RAM visible to the JVM (which could be all the physical RAM on
the machine if not run in a container-aware JVM). This commit therefore
sets the initial heap size to 4m, the max heap size to 64m, the garbage
collector to the serial collector, and leaves this user-configurable by
honoring ES_JAVA_OPTS last.
This adds a new dynamic cluster setting `xpack.data_frame.num_transform_failure_retries`.
This setting indicates how many times non-critical failures should be retried before a data frame transform is marked as failed and should stop executing. At the time of this commit; Min: 0, Max: 100, Default: 10
Several files in the REST APIs nav section are included using
:leveloffset: tags. This increments headings (h2 -> h3, h3 -> h4, etc.)
in those files and removes the :leveloffset: tags.
Other supporting changes:
* Alphabetizes top-level REST API nav items.
* Change 'indices APIs' heading to 'index APIs.'
* Changes 'Snapshot lifecycle management' heading to sentence case.
* We shouldn't just swallow the interrupt here quietly and keep going on the IO thread
* Currently interrupt continues here just the same way an invocation of `acceptIncomingRequests` woudl have made things continue
* Relates #44610
NamedAnalyzer should return the same AnalysisMode than any custom analyzer it
wraps, otherwise AnalysisMode.ALL. This used to be only CustomAnalyzer in the
past, but with the introduction of the ReloadableCustomAnalyzer this needs to be
added as an option where the analysis mode gets propagated.
Closes#44625
This commit removes support for the translog checkpoint format from versions
before 6.0.0 since 7.x versions are incompatible with indices from these
versions.
Relates #44720Fixes#44210
elastic/elasticsearch#41281 added custom metadata parameter to
snapshots. During review, the parameter name was changed from '_meta' to
'metadata,' but the documentation wasn't updated. This corrects the
documentation to use the 'metadata' name.
By mistake in 7.x types field was removed from slow logs. Types are
still present in that version, so this have to be present as a JSON
field
relates #41354
backport that was causing this #44178
This commit converts several utility classes that implement Streamable
to have StreamInput constructors. It also adds a default version of
readFrom to Streamable so that overriding to throw UOE is not necessary.
relates #34389
This commit converts several more classes from streamable to writeable
in server, mostly within the o.e.index and o.e.persistent packages.
relates #34389
* Allow empty configuration for SLM policies
When putting or updating a snapshot lifecycle policy it was not possible
to elide the `config` map. This commit makes the configuration optional,
the same way that it is when taking a snapshot.
Relates to #38461
* Add Objects.requireNonNull for required parts of the policy
* Expose index age in ILM explain output
This adds the index's age to the ILM explain output, for example:
```
{
"indices" : {
"ilm-000001" : {
"index" : "ilm-000001",
"managed" : true,
"policy" : "full-lifecycle",
"lifecycle_date" : "2019-07-16T19:48:22.294Z",
"lifecycle_date_millis" : 1563306502294,
"age" : "1.34m",
"phase" : "hot",
"phase_time" : "2019-07-16T19:48:22.487Z",
... etc ...
}
}
}
```
This age can be used to tell when ILM will transition the index to the
next phase, based on that phase's `min_age`.
Resolves#38988
* Expose age in getters and in HLRC
This commit converts subclasses of ShardOperationFailedException to
implement ctors with StreamInput instead of readFrom. It also simplifies
IndicesShardStoresResponse.Failure to serialize its shardId after the
super data.
relates #34389
Benjamin Trent
Armin Braun
Evgenia Badiyanova
David Turner
Jason Tedor
James Rodewig
Mayya Sharipova
Przemyslaw Gomulka
David Roberts
Alpar Torok
Ryan Ernst
Tal Levy
Christoph Büscher
Nikita Glashenko
Julie Tibshirani
Lisa Cawley
Lee Hinman