Today when we reset a recovery because of the source not being
ready or the shard is getting removed on the source (for whatever reason)
we wipe all temp files and reset the recovery without respecting any
reference counting or locking etc. all streams are closed and files are
wiped. Yet, this is problematic since we assert that some files are on disk
etc. when we finish writing a file. These assertions don't hold anymore if we
concurrently wipe the tmp files.
This change moves the logic out of RecoveryTarget into RecoveriesCollection which
basically clones the RecoveryTarget on reset instead which allows in-flight operations
to finish gracefully. This means we now have a single path for cleanups in RecoveryTarget
and can safely use assertions in the class since files won't be removed unless the recovery
is either canceled, failed or finished.
Closes #19473
Today they don't because the create index request that is implicitly created
adds an empty mapping for the type of the document. So to Elasticsearch it
looks like this type was explicitly created and `index.mapper.dynamic` is not
checked.
Closes#17592
The test would previously catch Throwable and then decide if it was a critical exception or not. As the catch block was changed from Throwable to Exception this made the test fail for non-critical exceptions. This commit changes the test so that exceptions are only thrown when they're unexpected.
This is the first pipeline aggregation that doesn't have its own
bucket type that needs serializing. It uses InternalHistogram instead.
So that required reworking the new-style `registerAggregation` method
to not require bucket readers. So I built `PipelineAggregationSpec` to
mirror `AggregationSpec`. It allows registering any number of bucket
readers or result readers.
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.
Preinstalled modules are:
* transport-netty
* lang-mustache
* reindex
* percolator
The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.
Closes#19412
Previously if the size of the search request was greater than zero we would not cache the request in the request cache.
This change retains the default behaviour of not caching requests with size > 0 but also allows the `request_cache=true` query parameter
to enable the cache for requests with size > 0
This is a tentative to revive #15939 motivated by elastic/beats#1941.
Half-floats are a pretty bad option for storing percentages. They would likely
require 2 bytes all the time while they don't need more than one byte.
So this PR exposes a new `scaled_float` type that requires a `scaling_factor`
and internally indexes `value*scaling_factor` in a long field. Compared to the
original PR it exposes a lower-level API so that the trade-offs are clearer and
avoids any reference to fixed precision that might imply that this type is more
accurate (actually it is *less* accurate).
In addition to being more space-efficient for some use-cases that beats is
interested in, this is also faster that `half_float` unless we can improve the
efficiency of decoding half-float bits (which is currently done using software)
or until Java gets first-class support for half-floats.
`create_bwc_index.py` has some cruft in it that only works on 2.x or
even before. This commit make the tool functional, yet there might still
be some bwc relevant things missing here.
Closes#19253
Today the default precision for the cardinality aggregation depends on how many
parent bucket aggregations it had. The reasoning was that the more parent bucket
aggregations, the more buckets the cardinality had to be computed on. And this
number could be huge depending on what the parent aggregations actually are.
However now that we run terms aggregations in breadth-first mode by default when
there are sub aggregations, it is less likely that we have to run the cardinality
aggregation on kagilions of buckets. So we could use a static default, which will
be less confusing to users.
We currently have concurrency issue between the static methods on the Store class and store changes that are done via a valid open store. An example of this is the async shard fetch which can reach out to a node while a local shard copy is shutting down (the fetch does check if we have an open shard and tries to use that first, but if the shard is shutting down, it will not be available from IndexService).
Specifically, async shard fetching tries to read metadata from store, concurrently the shard that shuts down commits to lucene, changing the segments_N file. this causes a file not find exception on the shard fetching side. That one in turns makes the master think the shard is unusable. In tests this can cause the shard assignment to be delayed (up to 1m) which fails tests. See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/570 for details.
This is one of the things #18938 caused to bubble up.
Maven central requires a project url. The recent change to make poms for
plugin client jars broke that because we no longer use nebula publishing
for plugin pom generation. This change adds back the url to the pom.
* Removed `Template` class and unified script & template parsing logic. Templates are scripts, so they should be defined as a script. Unless there will be separate template infrastructure, templates should share as much code as possible with scripts.
* Removed ScriptParseException in favour for ElasticsearchParseException
* Moved TemplateQueryBuilder to lang-mustache module because this query is hard coded to work with mustache only
This change removes the multiple ways that plugins can be added to the
integ test cluster. It also removes the use of the default
configuration, and instead adds a zip configuration to all plugins. This
will enable using project substitutions with plugins, which must be done
with the default configuration.
creation in the REST tests, as we no longer need it due
to index creation now waiting for active shard copies
before returning (by default, it waits for the primary of
each shard, which is the same as ensuring yellow health).
Relates #19450
This should make them easier to read and adds them to the test suite
I changed the example from a two node cluster to a single node cluster
because that is what we have running in the integration tests. It is also
what a user just starting out is likely to see so I think that is ok.
integration tests, as they are no longer needed with
index creation now waiting for shards to be started before
returning from the index creation call (by default, it waits
for the primary of each shard to be started before returning,
which is what ensureYellow() was ensuring anyway).
Closes#19452
Relates #19450
This commit adds a log message when bootstrap checks are enforced
informing the user that they are enforced because they are bound to an
external network interface. We also log if bootstrap checks are being
enforced but system checks are being ignored.
Relates #19451
The unit tests for IndicesClusterStateService currently inject random failures upon shard creation/ routing upate / mapping update etc. This commit makes injecting failures optional so that stronger assertions can be made about the local indices / shard state in case of no failures.
Before returning, index creation now waits for the configured number
of shard copies to be started. In the past, a client would create an
index and then potentially have to check the cluster health to wait
to execute write operations. With the cluster health semantics changing
so that index creation does not cause the cluster health to go RED,
this change enables waiting for the desired number of active shards
to be active before returning from index creation.
Relates #9126
For historical reasons, the value associated with Priority.IMMEDIATE is
-1. Yet, with a full-cluster restart required on major version upgrades,
we can reset these values so they are conceptually simpler. This commit
resets the values associated with Priority instances.
Today we have an abstraction Priority for representing
priorities. Ideally, these values are a fixed set of constants with a
well-defined ordering which sounds perfect for an enum. This commit
changes Priority so that it is an enum instead of a class.
In Priority there is a field named values that represents an ordered, by
priority, list of all priorities. Yet, this collection is modifiable and
this collection is exposed via the public API. This means that consumers
can modify this list potentially leading to complete chaos. This commit
modifies this field so that it is unmodifiable, documents that the
returned collection is unmodifiable, and returns total order to the
world. We also punish the bad consumer here by making them make a copy
of the returned collection with which they can do as they please. This
fixes a puzzling test failure which only arises if the two tests
(PrioritizedExecutorsTests#testPriorityQueue and
PriorityTests#testCompareTo run in the same JVM, and run in the right
order).
Relates #19447
Moving the dependent jars instead of copying breaks downstream builds
that rely on the jars existing for compilation. This commit modifies
these moves to be copies.
Also introduced a `Processor.Parameters` class that is holder for several services processors rely on,
the IngestPlugin#getProcessors(...) method has been changed to accept `Processor.Parameters` instead
of each service seperately.
Today we log all loaded modules and installed plugins in a single
line. The number of modules has grown, and when plugins are installed a
single log line containing the loaded modules and plugins is
lengthy. With this commit, we log a single module or plugin per line,
log these in sorted order, and also log if no modules or no plugins were
loaded.
Relates #19441
This commit renames the Netty 3 transport module from transport-netty to
transport-netty3. This is to make room for a Netty 4 transport module,
transport-netty4.
Relates #19439
Currently custom headers that should be passed through rest requests are
registered by depending on the RestController in guice and calling a
registration method. This change moves that registration to a getter for
plugins, and makes the RestController take the set of headers on
construction.
Simon Willnauer
Adrien Grand
Ryan Ernst
Karel Minarik
Nik Everett
Colin Goodheart-Smithe
Yannick Welsch
Martijn van Groningen
Boaz Leskes
Clinton Gormley
Ali Beyad
Jason Tedor
James Baiera