When starting a new replication group in an index level replication test
case, a started replica would not have a valid recovery state. This
violates simple assumptions as replicas always have to have recovered
before being started. This commit causes this to be the case that this
assumption is not violated too.
We previously removed this assertion because it could be violated in
races. This commit adds this assertion back with sampling done more
carefully to avoid failures solely due to race conditions.
When multiple bootstrap checks fail, it's not clear where one error
message begins and the next error message ends. This commit numbers the
bootstrap check error messages so they are easier to read.
Relates #24548
This starts breaking up the `UpdateHelper.prepare` method so that each piece can
be individually unit tested. No actual functionality has changed.
Note however, that I did add a TODO about `ctx.op` leniency, which I'd love to
remove as a separate PR if desired.
This commit fixes a bug in the cache expire after access
implementation. The bug is this: if you construct a cache with an expire
after access of T, put a key, and then touch the key at some time t > T,
the act of getting the key would update the access time for the entry
before checking if the entry was expired. There are situations in which
expire after access would be honored (e.g., if the cache needs to prune
the LRU list to keep the cache under a certain weight, or a manual
refresh was called) but this behavior is otherwise broken.
Relates #24546
In order to make MockLogAppender (utility to test logging) available outside
of es-core move MockLogAppender from test core-tests to test framework. As
package names do not change, no need to change clients.
Today when opening the engine we skip gaps in the history, advancing the
local checkpoint until it is equal to the maximum sequence number
contained in the commit. This allows history to advance, but it leaves
gaps. A previous change filled these gaps when recovering from store,
but since we were skipping the gaps while opening the engine, this
change had no effect. This commit removes the gap skipping when opening
the engine allowing the gap filling to do its job.
Relates #24535
I stumbled on this code today and I hated it; I wrote it. I did not like
that you could not tell at a glance whether or not the method parameters
were correct. This commit fixes it.
Relates #24522
This commit changes the Terms.Bucket abstract class to an interface, so
that it's easier for the Java High Level Rest Client to provide its own
implementation.
In its current state, the Terms.Bucket abstract class inherits from
InternalMultiBucketAggregation.InternalBucket which forces subclasses to
implement Writeable and exposes a public getProperty() method that relies
on InternalAggregation. This two points make it difficult for the Java
High Level Rest Client to implement the Terms and Terms.Bucket correctly.
This is also different from other MultiBucketsAggregation like Range
which are pure interfaces.
Changing Terms.Bucket to an interface causes a method clashes for the
`getBuckets()` method in InternalTerms. This is because:
- InternalTerms implements Terms which declared a
`List<Terms.Bucket> getBuckets()` method
- InternalTerms extends InternalMultiBucketAggregation which declares a
`List<? extends InternalBucket> getBuckets()` method
- both overrides the MultiBucketsAggregation
`List<? extends Bucket> getBuckets()` method
There was no clashes before this change because Terms.Bucket extends
InternalBucket and conformed to both declaration. With Terms.Bucket now
an interface, the getBuckets() method in the Terms interface is changed
to avoid method clash. This is a breaking change in the Java API but
it's a straightforward change and the Terms multi bucket aggregation
interface is also more coherent with the other Range, Histogram,
Filters, AdjacencyMatrix etc that all return a `List<? extends Bucket>`.
This commit adds support for indexing and searching a new ip_range field type. Both IPv4 and IPv6 formats are supported. Tests are updated and docs are added.
This change will expand the shard level request to the actual concrete index or to the aliases
that expanded to the concrete index to ensure shard level requests won't see wildcard expressions as their original indices
If a field caps request contains a field name that doesn't exist in all indices
the response will be partial and we hide an NPE. The NPE is now fixed but we still
have the problem that we don't pass on errors on the shard level to the user. This will
be fixed in a followup.
`_search_shards`API today only returns aliases names if there is an alias
filter associated with one of them. Now it can be useful to see which aliases
have been expanded for an index given the index expressions. This change also includes non-filtering aliases even without a filtering alias being present.
Changes the scope of the AllocationService dependency injection hack so that it is at least contained to the AllocationService and does not leak into the Discovery world.
Today we go to heroic lengths to workaround bugs in the JDK or around
issues like BSD jails to get information about the underlying file
store. For example, we went to lengths to work around a JDK bug where
the file store returned would incorrectly report whether or not a path
is writable in certain situations in Windows operating
systems. Another bug prevented getting file store information on
Windows on a virtual drive on Windows. We no longer need to work
around these bugs, we could simply try to write to disk and let an I/O
exception arise if we could not write to the disk or take advantage of
the fact that these bugs are fixed in recent releases of the JDK
(e.g., the file store bug is fixed since 8u72). Additionally, we
collected information about all file stores on the system which meant
that if the user had a stale NFS mount, Elasticsearch could hang and
fail on startup if that mount point was not available. Finally, we
collected information through Lucene about whether or not a disk was a
spinning disk versus an SSD, information that we do not need since we
assume SSDs by default. This commit takes into consideration that we
simply do not need this heroic effort, we do not need information
about all file stores, and we do not need information about whether or
not a disk spins to greatly simplfy file store handling.
Relates #24402
This commit fixes the reschedule async fsync test in index service
tests. This test was passing for the wrong reason. Namely, the test was
trying to set translog durability to async, fire off an indexing
request, and then assert that the translog eventually got fsynced. The
problem here is that in the course of issuing the indexing request, a
mapping update is trigger. The mapping update triggers the index
settings to be refreshed. Since the test did not issue a cluster state
update to change the durability from request to async but instead did
this directly through index service, the mapping update flops the
durability back to async. This means that when the indexing request
executes, an fsync is performed after the request and the assertoin that
an fsync is not needed passes but for the wrong reason (in short: the
test wanted it to pass because an async fsync fired, but instead it
passed because a request async fired). This commit fixes this by going
through the index settings API so that a proper cluster state update is
triggered and so the mapping update does not flop the durability back to
request.
We are still chasing a test failure here and increasing the logging
level stopped the failures. We have a theory as to what is going on so
this commit reduces the logging level to hopefully trigger the failure
again and give us the logging that we need to confirm the theory.
This is a follow-up to #24317, which did the hard work but was merged in such a
way that it exposes the setting while still allowing indices to have multiple
types by default in order to give time to people who test against master to
add this setting to their index settings.
This test started failing but the logging here is insufficient to
discern what is happening. This commit increases the logging level on
this test until the failure can be understood.
The UpgraderPlugin adds two additional extension points called during cluster upgrade and when old indices are introduced into the cluster state during initial recovery, restore from a snapshot or as a dangling index. One extension point allows plugin to update old templates and another extension points allows to update/check index metadata.
Currently the only implementation of `ListenableActionFuture` requires
dispatching listener execution to a thread pool. This commit renames
that variant to `DispatchingListenableActionFuture` and converts
`AbstractListenableActionFuture` to be a variant that does not require
dispatching. That class is now named `PlainListenableActionFuture`.
The DiscoveryNodesProvider interface provides an unnecessary abstraction and is just used in conjunction with the existing PingContextProvider interface. This commit removes it.
Open/Close index api have allow_no_indices set to false by default, while delete index has it set to true. The flag controls where a wildcard expression that matches no indices will be ignored or an error will be thrown instead. This commit aligns open/close default behaviour to that of delete index.
This field was only ever used in the constructor, where it was set and
then passed around. As such, there's no need for it to be a field and we
can remove it.
After a replica shard finishes recovery, it will be marked as active and
its local checkpoint will be considered in the calculation of the global
checkpoint on the primary. If there were operations in flight during
recovery, when the replica is activated its local checkpoint could be
lagging behind the global checkpoint on the primary. This means that
when the replica shard is activated, we can end up in a situtaion where
a global checkpoint update would want to move the global checkpoint
backwards, violating an invariant of the system. This only arises if a
background global checkpoint sync executes, which today is only a
scheduled operation and might be delayed until the in-flight operations
complete and the replica catches up to the primary. Yet, we are going to
move to inlining global checkpoints which will cause this issue to be
more likely to manifest. Additionally, the global checkpoint on the
replica, which is the local knowledge on the replica updated under the
mandate of the primary, could be higher than the local checkpoint on the
replica, again violating an invariant of the system. This commit
addresses these issues by blocking global checkpoint on the primary when
a replica shard is finalizing recovery. While we have blocked global
checkpoint advancement, recovery on the replica shard will not be
considered complete until its local checkpoint advances to the blocked
global checkpoint.
Relates #24404
today we only lookup nodes by their ID but never by the (clusterAlias, nodeId) tuple.
This could in theory lead to lookups on the wrong cluster if there are 2 clusters
with a node that has the same ID. It's very unlikely to happen but we now can clearly
disambiguate between clusters and their nodes.
This change makes the request builder code-path same as `Client#execute`. The request builder used to return a `ListenableActionFuture` when calling execute, which allows to associate listeners with the returned future. For async execution though it is recommended to use the `execute` method that accepts an `ActionListener`, like users would do when using `Client#execute`.
Relates to #24412
Relates to #9201
SearchResponseSections is the common part extracted from InternalSearchResponse that can be shared between high level REST and elasticsearch. The only bits left out are around serialization which is not supported. This way it can accept Aggregations as a constructor argument, without requiring InternalAggregations, as the high level REST client uses its own objects for aggs parsing rather than internal ones.
This change also makes Aggregations implement ToXContent, and Aggregation extend ToXContent. Especially the latter is suboptimal but the best solution that allows to share as much code as possible between core and the client, that doesn't require messing with generics and making the api complicated. Also it doesn't have downsides as all of the current implementations of Aggregation do implement ToXContent already.
Async shard fetching only uses the node id to correlate responses to requests. This can lead to a situation where a response from an earlier request is mistaken as response from a new request when a node is restarted. This commit adds unique round ids to correlate responses to requests.
We often want the JVM arguments used for a running instance of
Elasticsearch. It sure would be nice if these came as part of the nodes
API, or any API that includes JVM info. This commit causes these
arguments to be displayed.
Relates #24450
These can be very useful to have, let's have them at our fingertips in
the logs (in case the JVM dies and we can no longer retrieve them, at
least they are here).
Relates #24451
We've had `QueryDSLDocumentationTests` for a while but it had a very
hopeful comment at the top about how we want to make sure that the
example in the query-dsl docs match up with the test but we never
had anything that made *sure* that they did. This changes that!
Now the examples from the query-dsl docs are all built from the
`QueryDSLDocumentationTests`. All except for the percolator example
because that is hard to do as it stands now.
To make this easier this change moves `QueryDSLDocumentationTests`
from core and into the high level rest client. This is useful for
two reasons:
1. We expect the high level rest client to be able to use the builders.
2. The code that builds that docs doesn't check out all of
Elasticsearch. It only checks out certain directories. Since we're
already including snippets from that directory we don't have to
make any changes to that process.
Closes#24320
TransportService and RemoteClusterService are closely coupled already today
and to simplify remote cluster integration down the road it can be a direct
dependency of TransportService. This change moves RemoteClusterService into
TransportService with the goal to make it a hidden implementation detail
of TransportService in followup changes.
RemoteClusterService is an internal service that should not necessarily be exposed
to plugins or other parts of the system. Yet, for cluster name parsing for instance
it is crucial to reuse some code that is used for the RemoteClusterService. This
change extracts a base class that allows to share the settings related code as well
as cluster settings updates to `search.remote.*` to be observed by other services.
