Today when a node is removed the cluster (it leaves or it fails), we
submit a cluster state update task. These cluster state update tasks are
processed serially on the master. When nodes are removed en masse (e.g.,
a rack is taken down or otherwise becomes unavailable), the master will
be slow to process these failures because of the resulting reroutes and
publishing of each subsequent cluster state. We improve this in this
commit by processing the node removals using the cluster state update
task batch processing framework.
Relates #19289
Today we have a bunch of tests that use netty transport for several reasons
these tests use it because they need to run some tcp based transport. Yet, this
couples our tests tightly to the netty implementation which should be tested on it's own.
This change adds a plain socket based blocking TcpTransport implementation that is used by
default in tests if local transport is suppressed or if network is selected.
It also adds another tcp network implementation as a showcase how the interface works.
Lucene IndexWriter asserts on files existing on the filesystem but
some tests throw IOException explicitly on those operatiosn such that
some tests trip asserts. We had this before on InternalEngine#ctor
and added some logic there to catch only a specific assertions based
on some excepition stack analysis. This change applies the same logic
to the IndexWriter#commit part of the engine since it can hit the same
issue.
This also fixes a self-suppression issue in Store.java.
Closes#19356
Several tests required http.enabled where it was unnecessary.
We also had RestMainActionIT which tests what two of our REST tests
test already so I removed it.
The explicit use of http.enabled: false is also obsolet since our
test do that by default.
The DiscoveryNodeService exists to register CustomNodeAttributes which
plugins can add. This is not necessary, since plugins can already add
additional attributes, and use the node attributes prefix.
This change removes the DiscoveryNodeService, and converts the only
consumer, the ec2 discovery plugin, to add the ec2 availability zone
in additionalSettings().
This change removes the ability for guice to have child injectors (and
the entire concept of parent injectors) from our fork of guice. The
methodology for removing was simple: I removed createChildInjector, and
continued to remove methods and members that were unused until my head
was spinning. The motivation for this change is to limit what our fork
of guice gives us access to, so we don't regress and start adding back
more complicated uses.
This adds a test that uses transport implementation and sends random requests
to 3 different nodes, the request handlers maybe forwarding the requests to yet another node
etc. until returning the response. This test basically tests that nodes are not deadlocking
in a distributed fashion.
Repository plugins currently use a lot of custom classes like
RepositoryName and RepositorySettings in order to use guice to construct
repository implementations. But repositories now only really need their
settings to be constructed. Anything else they need (eg a cloud client)
can be constructed within the plugin, instead of via guice.
This change makes repository plugins use the new pull model. It removes
guice from the construction of Repository objects (no more child
injectors) and also from all repository plugins.
This exposes a method to start an action and return a task from
`NodeClient`. This allows reindex to use the injected `Client` rather
than require injecting `TransportAction`s
Today when a thread encounters a fatal unrecoverable error that
threatens the stability of the JVM, Elasticsearch marches on. This
includes out of memory errors, stack overflow errors and other errors
that leave the JVM in a questionable state. Instead, the Elasticsearch
JVM should die when these errors are encountered. This commit causes
this to be the case.
Relates #19272
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.
This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests to integration tests.
This commit moves back some messy tests that have been placed in lang-groovy module in https://github.com/elastic/elasticsearch/pull/13834. It removes the dependency on Groovy plugin as well as change back the tests to integration tests (IT suffix).
It also changes the current MockScriptEngine and MockScriptPlugin to make it easier to use.
The api for snapshot/restore was split up between two interfaces,
Repository and IndexShardRepository. There was also complex
initialization and injection between the two. However, there is always a
one to one relationship between the two.
This change moves the IndexShardRepository api into Repository, as well
as updates the API so as not to require any services to be injected for
sublcasses.
This adds a new proxy for RestHandlers and RestControllers so that requests made
to deprecated REST APIs can be automatically logged in the ES logs via the
DeprecationLogger as well as via a "Warning" header (RFC-7234) for all responses.
Calling indicesService.deleteIndex() can trip an assertion if there is an ongoing cluster state applied in
IndicesClusterStateService. This means that the index is possibly deleted after the failMissingShards
check and before we try creating new and updated shards, tripping an assertion that non-existing shards must
have shard state initializing (started in this case).
This change adds the type of the field in the fieldstats response.
It can be one of the following:
* "integer" for byte, short, integer and long
* "float" for float, half-float and double
* "date" for date
* "ip" for ip
* "text" for string, keyword and text.
Closes#17750
This adds a remote option to reindex that looks like
```
curl -POST 'localhost:9200/_reindex?pretty' -d'{
"source": {
"remote": {
"host": "http://otherhost:9200"
},
"index": "target",
"query": {
"match": {
"foo": "bar"
}
}
},
"dest": {
"index": "target"
}
}'
```
This reindex has all of the features of local reindex:
* Using queries to filter what is copied
* Retry on rejection
* Throttle/rethottle
The big advantage of this version is that it goes over the HTTP API
which can be made backwards compatible.
Some things are different:
The query field is sent directly to the other node rather than parsed
on the coordinating node. This should allow it to support constructs
that are invalid on the coordinating node but are valid on the target
node. Mostly, that means old syntax.
This commit renames writeThrowable to writeException. The situation here
stems from the fact that the StreamOutput method for serializing
Exceptions needs to accept Throwables too as Throwables can be the cause
of serialized Exceptions. Yet, we do not serialize Throwables in the
Error sub-hierarchy in a way that they can be deserialized into their
initial type. This leads to an asymmetry in the StreamOutput method for
serializing Exceptions and the StreamInput method for writing
Excpetions. Namely, the former will accept Throwables but the latter
will only return Exceptions. A goal with the stream methods has always
been symmetry in the method names so that serialization/deserialization
routines appear symmetrical in code. It is this asymmetry on the
input/output types for Exceptions on StreamOutput/StreamInput that
clashes with the desired symmetry of naming. Despite this, we should
favor symmetry in the naming of the methods. This commit renames
StreamOutput#writeThrowable to StreamOutput#writeException which leaves
us with Exception StreamInput#readException and void
StreamOutput#writeException(Throwable).
This commit modifies the initial value of the transport client
round-robin index to a random value so that initial requests are more
likely to not all hit the same node.
Relates #14143
Due to some optimization on the netty layer we had quite some code / cruft
added to the TcpTransport to allow for those optimizations. After cleaning
up BytesReference we can now move this optimization into TcpTransport and
have a simple send method on the implementation layer instead. This commit
adds a CompositeBytesReference that also allows message headers to be written
separately which simplify the header code as well since no skips are needed
anymore.
Jason Tedor
Simon Willnauer
javanna
Ryan Ernst
Nicholas Knize
Martijn van Groningen
Simon Willnauer
Nik Everett
Tanguy Leroux
Clinton Gormley
Alexander Reelsen
Chris Earle
Yannick Welsch
Jim Ferenczi
Igor Motov
Justin Patrin
LeeDr
Adrien Grand