RecoveryTarget increments a reference on the store once it's
created. If we fail to return the instance from the reset method
we leak a reference causing shard locks to not be released. This
change creates the reference in the return statement to ensure no
references are leaked
An initializing replica shard might not have an UnassignedInfo object, for example when it is a relocation target. The method allocatedPostIndexCreate does not account for this situation.
Today when we reset a recovery because of the source not being
ready or the shard is getting removed on the source (for whatever reason)
we wipe all temp files and reset the recovery without respecting any
reference counting or locking etc. all streams are closed and files are
wiped. Yet, this is problematic since we assert that some files are on disk
etc. when we finish writing a file. These assertions don't hold anymore if we
concurrently wipe the tmp files.
This change moves the logic out of RecoveryTarget into RecoveriesCollection which
basically clones the RecoveryTarget on reset instead which allows in-flight operations
to finish gracefully. This means we now have a single path for cleanups in RecoveryTarget
and can safely use assertions in the class since files won't be removed unless the recovery
is either canceled, failed or finished.
Closes #19473
Today they don't because the create index request that is implicitly created
adds an empty mapping for the type of the document. So to Elasticsearch it
looks like this type was explicitly created and `index.mapper.dynamic` is not
checked.
Closes#17592
The test would previously catch Throwable and then decide if it was a critical exception or not. As the catch block was changed from Throwable to Exception this made the test fail for non-critical exceptions. This commit changes the test so that exceptions are only thrown when they're unexpected.
This is the first pipeline aggregation that doesn't have its own
bucket type that needs serializing. It uses InternalHistogram instead.
So that required reworking the new-style `registerAggregation` method
to not require bucket readers. So I built `PipelineAggregationSpec` to
mirror `AggregationSpec`. It allows registering any number of bucket
readers or result readers.
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.
Preinstalled modules are:
* transport-netty
* lang-mustache
* reindex
* percolator
The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.
Closes#19412
Previously if the size of the search request was greater than zero we would not cache the request in the request cache.
This change retains the default behaviour of not caching requests with size > 0 but also allows the `request_cache=true` query parameter
to enable the cache for requests with size > 0
This is a tentative to revive #15939 motivated by elastic/beats#1941.
Half-floats are a pretty bad option for storing percentages. They would likely
require 2 bytes all the time while they don't need more than one byte.
So this PR exposes a new `scaled_float` type that requires a `scaling_factor`
and internally indexes `value*scaling_factor` in a long field. Compared to the
original PR it exposes a lower-level API so that the trade-offs are clearer and
avoids any reference to fixed precision that might imply that this type is more
accurate (actually it is *less* accurate).
In addition to being more space-efficient for some use-cases that beats is
interested in, this is also faster that `half_float` unless we can improve the
efficiency of decoding half-float bits (which is currently done using software)
or until Java gets first-class support for half-floats.
Today the default precision for the cardinality aggregation depends on how many
parent bucket aggregations it had. The reasoning was that the more parent bucket
aggregations, the more buckets the cardinality had to be computed on. And this
number could be huge depending on what the parent aggregations actually are.
However now that we run terms aggregations in breadth-first mode by default when
there are sub aggregations, it is less likely that we have to run the cardinality
aggregation on kagilions of buckets. So we could use a static default, which will
be less confusing to users.
We currently have concurrency issue between the static methods on the Store class and store changes that are done via a valid open store. An example of this is the async shard fetch which can reach out to a node while a local shard copy is shutting down (the fetch does check if we have an open shard and tries to use that first, but if the shard is shutting down, it will not be available from IndexService).
Specifically, async shard fetching tries to read metadata from store, concurrently the shard that shuts down commits to lucene, changing the segments_N file. this causes a file not find exception on the shard fetching side. That one in turns makes the master think the shard is unusable. In tests this can cause the shard assignment to be delayed (up to 1m) which fails tests. See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/570 for details.
This is one of the things #18938 caused to bubble up.
* Removed `Template` class and unified script & template parsing logic. Templates are scripts, so they should be defined as a script. Unless there will be separate template infrastructure, templates should share as much code as possible with scripts.
* Removed ScriptParseException in favour for ElasticsearchParseException
* Moved TemplateQueryBuilder to lang-mustache module because this query is hard coded to work with mustache only
integration tests, as they are no longer needed with
index creation now waiting for shards to be started before
returning from the index creation call (by default, it waits
for the primary of each shard to be started before returning,
which is what ensureYellow() was ensuring anyway).
Closes#19452
Relates #19450
This commit adds a log message when bootstrap checks are enforced
informing the user that they are enforced because they are bound to an
external network interface. We also log if bootstrap checks are being
enforced but system checks are being ignored.
Relates #19451
The unit tests for IndicesClusterStateService currently inject random failures upon shard creation/ routing upate / mapping update etc. This commit makes injecting failures optional so that stronger assertions can be made about the local indices / shard state in case of no failures.
Before returning, index creation now waits for the configured number
of shard copies to be started. In the past, a client would create an
index and then potentially have to check the cluster health to wait
to execute write operations. With the cluster health semantics changing
so that index creation does not cause the cluster health to go RED,
this change enables waiting for the desired number of active shards
to be active before returning from index creation.
Relates #9126
For historical reasons, the value associated with Priority.IMMEDIATE is
-1. Yet, with a full-cluster restart required on major version upgrades,
we can reset these values so they are conceptually simpler. This commit
resets the values associated with Priority instances.
Today we have an abstraction Priority for representing
priorities. Ideally, these values are a fixed set of constants with a
well-defined ordering which sounds perfect for an enum. This commit
changes Priority so that it is an enum instead of a class.
In Priority there is a field named values that represents an ordered, by
priority, list of all priorities. Yet, this collection is modifiable and
this collection is exposed via the public API. This means that consumers
can modify this list potentially leading to complete chaos. This commit
modifies this field so that it is unmodifiable, documents that the
returned collection is unmodifiable, and returns total order to the
world. We also punish the bad consumer here by making them make a copy
of the returned collection with which they can do as they please. This
fixes a puzzling test failure which only arises if the two tests
(PrioritizedExecutorsTests#testPriorityQueue and
PriorityTests#testCompareTo run in the same JVM, and run in the right
order).
Relates #19447
Also introduced a `Processor.Parameters` class that is holder for several services processors rely on,
the IngestPlugin#getProcessors(...) method has been changed to accept `Processor.Parameters` instead
of each service seperately.
Today we log all loaded modules and installed plugins in a single
line. The number of modules has grown, and when plugins are installed a
single log line containing the loaded modules and plugins is
lengthy. With this commit, we log a single module or plugin per line,
log these in sorted order, and also log if no modules or no plugins were
loaded.
Relates #19441
This commit renames the Netty 3 transport module from transport-netty to
transport-netty3. This is to make room for a Netty 4 transport module,
transport-netty4.
Relates #19439
Currently custom headers that should be passed through rest requests are
registered by depending on the RestController in guice and calling a
registration method. This change moves that registration to a getter for
plugins, and makes the RestController take the set of headers on
construction.
Invocation counts can be used to help judge the selectivity of individual query components in the context of the entire query. E.g. a query may not look selective when run by itself (matches most of the index), but when run in context of a full search request, is evaluated only rarely due to execution order
Since this is modifying the base timing class, it'll enrich both query and agg profiles (as well as future profile results)
Today `node.mode` and `node.local` serve almost the same purpose, they
are a shortcut for `discovery.type` and `transport.type`. If `node.local: true`
or `node.mode: local` is set elasticsearch will start in _local_ mode which means
only nodes within the same JVM are discovered and a non-network based transport
is used. The _local_ mode it only really used in tests or if nodes are embedded.
For both, embedding and tests explicit configuration via `discovery.type` and `transport.type`
should be preferred.
This change removes all the usage of these settings and by-default doesn't
configure a default transport implemenation since netty is now a module. Yet, to make
the user expericence flawless, plugins or modules can set a `http.type.default` and
`transport.type.default`. Plugins set this via `PluginService#additionalSettings()`
which enforces _set-once_ which prevents node startup if set multiple times. This means
that our distributions will just startup with netty transport since it's packaged as a
module unless `transport.type` or `http.transport.type` is explicitly set.
This change also found a bunch of bugs since several NamedWriteables were not registered if a
transport client is used. Now that we don't rely on the `node.mode` leniency which is inherited
instead of using explicit settings, `TransportClient` uses `AssertingLocalTransport` which detects these problems since it serializes all messages.
Closes#16234
The Java API supports this while mostly used for tests it can also be useful in
production environments. For instance if something is automated like a settings change
and we execute some health right after it the settings update might have some consequences
like a reroute which hasn't been fully applied since the preconditions are not fulfilled yet.
For instance if not all shards started the settings update is applied but the reroute won't move
currently initializing shards like in the shrink API test. Sure this could be done by waiting for
green before but if the cluster moves shards due to some side-effects waiting for all events is
still useful. I also took the chance to add unittests to Priority.java
Closes#19419
This adds an extra method, registerWithDeprecatedHandler, to register both a normal handler and a deprecated handler at the same time. This helps with renaming methods as opposed to _just_ deprecated methods.
Today we assert that the tmp files are present but if the recovery
was canceled this might not be the case while still a valid state.
This chance only throws the AssertionError if the recovery is still active.
This moves all netty related code into modules/transport-netty the module is build as a zip file as well as a JAR to serve as a dependency for transport client. For the time being this is required otherwise we have no network based impl. for transport client users. This might be subject to change given that we move forward http client.
Simon Willnauer
Martijn van Groningen
Yannick Welsch
Simon Willnauer
Adrien Grand
Nik Everett
Colin Goodheart-Smithe
Boaz Leskes
Ali Beyad
Jason Tedor
Ryan Ernst
Zachary Tong
Mathias Fussenegger
Tal Levy
gfyoung
Chris Earle