This commit modifies the behavior after publication of a new cluster
state to only invoke the reroute logic once per batch of shard failures
rather than once per shard failure.
This commit modifies the internal representation of the JVM flag
UseCompressedOops to just be a String. This means we can just store the
value of the flag or "unknown" directly so that we do not have to engage
in shenanigans with three-valued logic around a boxed boolean.
Relates #15489
With this commit we introduce limited retries with a backoff logic to BulkProcessor
when a bulk request has been rejeced with an EsRejectedExecutionException.
Fixes#14620.
Currently MetaDataMappingService parses the mapping updates, reserializes it and
finally calls MapperService.merge with the serialized mapping. Given that mapping
serialization only writes differences from the default, this is a bit unfair to
parsers since they can't know whether some option has been explicitly set or not.
Furthermore this can cause bugs with metadata fields given that these fields use
existing field types as defaults.
This commit changes MetaDataMappingService to call MapperService.merge with the
original mapping update.
This fixes the `lenient` parameter to be `missingClasses`. I will remove this boolean and we can handle them via the normal whitelist.
It also adds a check for sheisty classes (jar hell with the jdk).
This is inspired by the lucene "sheisty" classes check, but it has false positives. This check is more evil, it validates every class file against the extension classloader as a resource, to see if it exists there. If so: jar hell.
This jar hell is a problem for several reasons:
1. causes insanely-hard-to-debug problems (like bugs in forbidden-apis)
2. hides problems (like internal api access)
3. the code you think is executing, is not really executing
4. security permissions are not what you think they are
5. brings in unnecessary dependencies
6. its jar hell
The more difficult problems are stuff like jython, where these classes are simply 'uberjared' directly in, so you cant just fix them by removing a bogus dependency. And there is a legit reason for them to do that, they want to support java 1.4.
We currently randomly add a set of mock plugins to integ tests.
Sometimes it is necessary to omit this mock plugins, but other times you
may just want to suppress a particular mock plugin. For example, if you
have your own transport, you want to omit the asserting local transport
mock, since they would both try to set the transport.type.
This commit adds a callback for a cluster state task executor that will
be invoked if the execution of a batch of cluster state update tasks
led to a new cluster state and that new cluster state was successfully
published.
Closes#15482
This commit adds to JvmInfo the status of whether or not compressed
ordinary object pointers are enabled. Additionally, logging of the max
heap size and the status of the compressed ordinary object pointers flag
are provided on startup.
Relates #13187, relates elastic/elasticsearch-definitive-guide#455
When a client sends a request to fail a shard to the master, the current
behavior is that the master will submit the cluster state update task
and then immediately send a successful response back to the client;
additionally, if there are any failures while processing the cluster
state update task to fail the shard, then the client will never be
notified of these failures.
This commit modifies the master behavior when handling requests to fail
a shard. In particular, the master will now wait until successful
publication of the cluster state update before notifying the request
client that the shard is marked as failed; additionally, the client is
now notified of any failures during the execution of the cluster state
update task.
Relates #14252
When creating a metadata mapper for a new type, we reuse an existing
configuration from an existing type (if any) in order to avoid introducing
conflicts. However this field type that is provided is considered as both an
initial configuration and the default configuration. So at serialization time,
we might only serialize the difference between the current configuration and
this default configuration, which might be different to what is actually
considered the default configuration.
This does not cause bugs today because metadata mappers usually override the
toXContent method and compare the current field type with Defaults.FIELD_TYPE
instead of defaultFieldType() but I would still like to do this change to
avoid future bugs.
Today we only test this when writing sequentially. Yet, in practice we mainly
write concurrently, this commit adds a test that tests that concurrent writes with
sudden fatal failure will not corrupt our translog.
Relates to #15420
