Setting realTime to false in the get term vector request ensures,
that after a refresh the document is not fetched from the translog,
which seems to yield different in rare test runs.
The change likely triggering this was introduced in #15933
From this commit on we also validate all settings starting with `index.` that are
node-level settings configured in yaml files or via commandline arguments or system properties.
This check happens on node startup before the actual node is started.
The old infa has been removed in this commit such that nothing uses `DynamicSettings` anymore
and all index-scoped settings require to be registered before the node has fully started up.
This commit adds some additional assertions that test success is not
falsely indicated by adding assertions that success / failure methods
are not incorrectly invoked in failure / success scenarios.
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
This commit adds a test for calculating the age of PrioritizedRunnable
that allows real clock time to elapse. The test ensures that at least
one millisecond has passed, and that the resolution of System#nanoTime
on the underlying system is actually able to detect this.
Relates #15995
This commit wraps a trace logging message in a trace logging level check
to prevent allocating an Object array (to hold the logging parameters)
and a String (from the interval) when trace logging is not enabled every
second (with the default index refresh interval) and every five seconds
(with the default translog sync interval) for every open index when
trace logging is not enabled.
This commit simplifies an equality test in IndexShard#sameException
where the messages for two exceptions are being compared. The previous
condition first tested logical equality if the left exception is not
null, and otherwise tested reference equality. There is a convenience
method since JDK 7 for testing equality in this way: Objects#equals.
Closes#16025
Fixes an issue caused by trying to add a LeafReader for a closed index to
ShardCoreKeyMap. It add itself to the map half way before throwing
AlreadyClosedException, leaking some references and causesing Elasticsearch
to refuse to shut down cleanly when testing.
This commit enhances the master channel exception test in
o.e.c.a.s.ShardStateActionTests to test that a retries loop as expected
when requests to the master repeatedly fail.
When a metadata mapper is not specified in a mapping update, it should default
to the current metadata mapper instead of the general default in order for the
update to not conflict with the current mapping.
Closes#15997
It had some funky errors, like lenient:true not working and queries with
two integer fields blowing up if there was no analyzer defined on the
query. This throws a bunch more tests at it and rejiggers how non-strings
are handled so they don't wander off into scary QueryBuilder-land unless
they have a nice strong analyzer to protect them.
This commit adds a trace log on a cluster state update while waiting for
a new master, and changes the log level on cluster service close to the
warn level.
This commit tightens the tests in o.e.c.a.s.ShardStateActionTests:
- adds a simple test for a success condition that validates the shard
failed request is correct and sent to the correct place
- remove redundant assertions from the no master and master left tests
- an assertion that success is not falsely indicated in the case of a
unhandled error
the environment is now available through NodeModule#getNode#getEnvironment and can be retrieved during onModule(NodeModule), no need for this indirection anymore using the BiFunction
This commit addresses a time unit conversion bug in calculating the age
of a PrioritizedRunnable. The issue was an incorrect conversion from
nanoseconds to milliseconds as instead the conversion was to
microseconds. This leads to the timeInQueue metric for pending tasks to
be off by three orders of magnitude.
* Folded IngestModule into NodeModule
* Renamed IngestBootstrapper to IngestService
* Let NodeService construct IngestService and removed the Guice annotations
* Let IngestService implement Closable
We have a performance bug that if a filter aggregation is below a terms
aggregation that has a cardinality of 1000, we will call Query.createWeight
1000 times as well. However, Query.createWeight can be a costly operation.
For instance in the case of a TermQuery it will seek the term in every
segment. Instead, we should create the Weight once, and then get as many
iterators as we need from this Weight.
I found this problem while trying to diagnose a performance regression while
upgrading from 1.7 to 2.1[1]. While the problem was not introduced in 2.x, the
fact that 1.7 cached very aggressively had hidden this problem, since you don't
need to seek the term anymore on a cached TermFilter.
Doing things once for every aggregator is not easy with the current API but
I discussed this with Colin and Aggregator factories will need to get an init
method for different reasons, where we will be able to put these steps that
need to be performed only once, no matter haw many aggregators need to be
created.
[1] https://discuss.elastic.co/t/aggregations-in-2-1-0-much-slower-than-1-6-0/38056/26
If an operating system reports -1 for the total bytes of a filesystem
path, we should ignore it when capturing the least and most available
statistics.
Relates to #15919
Squashed commit of the following:
commit 5d2258ffeff8a0d156295dcc754ab9b6cbb4b02e
Author: Lee Hinman <lee@writequit.org>
Date: Thu Jan 14 14:14:27 2016 -0700
Change test to test positive total with negative 'free' value
commit 927e61d4b39692fc147220a955b63b291ad80db5
Author: Lee Hinman <lee@writequit.org>
Date: Thu Jan 14 13:09:28 2016 -0700
Skip capturing least/most FS info for an FS with no total
If an operating system reports -1 for the total bytes of a filesystem
path, we should ignore it when capturing the least and most available
statistics.
Relates to #15919
This commit removes the timeout retry mechanism from ShardStateAction
allowing it to instead be handled by the general master channel retry
mechanism. The idea is that if there is a network issue, the master will
miss a ping timeout causing the channel to be closed which will expose
itself via a NodeDisconnectedException. At this point, we can just wait
for a new master and retry, as with any other master channel exception.
This commit moves the handling of channel failures when failing a shard
to o.e.c.a.s.ShardStateAction. This means that shard failure requests
that timeout or occur when there is no master or the master leaves after
the request is sent will now be retried from here. The listener for a
shard failed request will now only be notified upon successful
completion of the shard failed request, or when a catastrophic
non-channel failure occurs.
This commit adds a simulation of the master leaving after a shard
failure request has been sent. In this case, after a new cluster state
is published (simulating a new master having been elected), the request
to fail the shard should be retried.
This commit handles the situation when we are failing a shard and either
no master is known, or the known master left while failing the shard. We
handle this situation by waiting for a new master to be reelected, and
then sending the shard failed request to the new master.
To main concern with the dedicated ingest TP is that there are already many TPs and in the case with beefy nodes we would many more threads. In the case ingest isn't used the all these threads are just idle.
Adding serialization capabilities to RescoreBuilder and make
all QueryRescorer implement NamedWritable, also requiring
all implementations of RescoreBuilder.Rescorer to implement
equals() and hashCode.
In addition, the current rescore mode enumeration is pulled out to a
separate class to make sharing of constants easier between
the query builders XContent rendering coder and the parser.
This change affects get alias, get aliases as well as cat aliases. They all return closed indices too by default. get alias and get aliases also allow to return open indices only through the `expand_wildcards` option (set it to `open`).
Closes#14982
This undocumented setting was mainly used for testing and a safety net for
a new flushOnClose feature back in 1.x. We can now safely remove this setting.
The test usage now uses a mock plugin setting to achive the same.
Today we maintain a lot of settings on the shard level which are all index level settings.
In order to cut over to the new settings API where we register update listener we have to move
all of them on to the index level otherwise we need a way to un-register listeners which is error-prone
and requires additional handling when shards are closed. It's simpler and also more accurate to handle all of
them on the index level where we can trash the entire registry for update listener once the index goes out of scope.
Don't set the suppressed Exception in Translog.closeOnTragicEvent(Exception ex) if it is an
AlreadyClosedException. ACE is thrown by the TranslogWriter and as cause might
contain the Exception that we add the suppressed ACE to. We then end up with a
circular reference where Exception A has a suppressed Exception B that has as cause A.
This would cause a stackoverflow when we try to serialize it.
For a more detailed description see #15941closes#15941
This commit adds load averages to the OS stats on FreeBSD. For these
stats to be available, linprocfs must be available and mounted at
/compat/linux/proc.
`refresh_interval` is a per index setting but we interpret and maintain it per shard. This
change moves the refresh task outside of IndexShard to the IndexService where it logically belongs
and reuses scheduling infrastructure used for translog fsync (async commit).
This change will use the same task for all shards of an index while previously we used on thread/task
per shard to refresh. This will also prevent too many concurrent refreshes if there are many indices and
shards allocated on a single node.
These filters leak into highlighting and probably other places and cause
things like the type name to be highlighted when using
requireFieldMatch=false. We could have special hacks to keep them out of
highlighting but it feals better to keep them out of any variable named
"originalQuery".
Closes#15689
This commit adds convenience methods to o.e.t.t.CapturingTransport
that enables capturing requests and clearing the captured requests
with a single method. This is to simplify a common pattern in tests of
capturing requests, and then clearing the captured requests.
Also renaming internal methods to reflect that they are dealing with
jts coordinates. Also renamed the list() to build() method for creating
the coordinates lists and adding constructors to PolygonBuilder that
take CoordinatesBuilders and implicitely call build() on them.
Rather than failing the request, when a node with node.ingest set to false receives an index or bulk request with a pipeline id, it should try to redirect the request to another node with node.ingest set to true. If there are no node with ingest set to true based on the current cluster state, an exception will be returned and the request will fail. Note that in case there are no ingest nodes and bulk has a pipeline id specified only for a subset of index requests, the whole bulk will fail.
The indexing buffer on a node (default: 10% of the JVM heap) is now a "shared pool" across all shards on that node. This way, shards doing intense indexing can use much more than other shards doing only light indexing, and only once the sum of all indexing buffers across all shards exceeds the node's indexing buffer will we ask shards to move recently indexed documents to segments on disk.
determined, the UpdateRequest would still try to parse the content instead
of throwing the standard ElasticsearchParseException. This manifests when
passing illegal JSON in the request body that does not begin with a '{'.
By trying to parse the content from an unknown request body content type,
the UpdateRequest was throwing a null pointer exception. This has been
fixed to throw an ElasticsearchParseException, to be consistent with the
behavior of all other requests in the face of undecipherable request
content types.
Closes#15822
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.
I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.
Closes#15847
So far the validation of geo shapes was only taking place in the
parse methods in ShapeBuilder. With the recent refactoring we no
longer can rely on shapes being parsed from json, so the same kind
of validation should take place when just using the java api.
A lot of validation concerns the number of points a shape needs to
have in order to be valid. Since this is not possible with current
builders where points can be added one by one, the builder constructors
are changed to require the mandatory parameters and validate those
already at construction time. To help with constructing longer lists
of points, a new utility PointsListBuilder is instroduces which can
produce list of coordinates accepted by most of the other shape builder
constructors.
Also adding tests for invalid shape exceptions to the already existing
shape builder tests.
Now that the ingest infra is part of es core we can remove some code that was required by the plugin and have a better integration with es core. We allow to specify the pipeline id in bulk and index as a request parameter, we have a REST filter that parses it and adds it to the relevant action request. That is not required anymore, as we can add this logic to RestIndexAction and RestBulkAction directly, no need for a filter. Also, we can allow to specify a pipeline id for each index requests in a bulk request. The small downside of this is that the ingest filter has to go over each item of a bulk request, all the time, to figure out whether they have a pipeline id.
Running requests via the percolate or mpercolate api is irrelevant.
What is relevant is that when nodes come back that they report the expected number of matches.
We fail today with ClusterBlockExceptions if an alias expands to a closed index
during search since we miss to check the index option down the road after we expanded
aliases.
Closes#13278
Nik Everett
Isabel Drost-Fromm
Simon Willnauer
Martijn van Groningen
Alexander Reelsen
Britta Weber
Jason Tedor
Ryan Ernst
Tal Levy
Adrien Grand
javanna
Lee Hinman
Luca Cavanna
Christoph Büscher
Simon Willnauer
Britta Weber
Michael McCandless
Ali Beyad
Michael McCandless