This commit removes the RequestBuilder generic type from Action. It was
needed to be used by the newRequest method, which in turn was used by
client.prepareExecute. Both of these methods are now removed, along with
the existing users of prepareExecute constructing the appropriate
builder directly.
This change deprecates completion queries and documents without context that target a
context enabled completion field. Querying without context degrades the search
performance considerably (even when the number of indexed contexts is low).
This commit targets master but the deprecation will take place in 6.x and the functionality
will be removed in 7 in a follow up.
Closes#29222
This commit adds Verify Repository, the associated docs and tests for
the high level REST API client. A few small changes to the Verify
Repository Response went into the commit as well.
Relates #27205
Currently failures to compile a script usually lead to a ScriptException, which
inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does
not contain another root cause. Instead, this should be a 400 Bad Request error.
This PR changes this more generally for script compilation errors by changing
ScriptException to return 400 (bad request) as status code.
Closes#12315
When we are connecting to a remote cluster we should never select
dedicated master nodes as gateway nodes, or we will end up loading them
with requests that should rather go to other type of nodes e.g. data
nodes or coord_only nodes.
This commit adds the selection based on the node role, to the existing
selection based on version and potential node attributes.
Closes#30687
AliasMetaData should be parsed more leniently so that the high-level REST client can support forward compatibility on it. This commit addresses this issue that was found as part of #28799 and adds dedicated XContent tests as well.
With multiple data paths, we write the state files for index metadata to all data paths. We only properly fsync on the first location, though. For other locations, we possibly expose the file before its contents is properly fsynced. This can lead to situations where, after a crash, and where the first data path is not available anymore, ES will see a partially-written state file, preventing the node to start up.
This change adds a new option to the composite aggregation named `missing_bucket`.
This option can be set by source and dictates whether documents without a value for the
source should be ignored. When set to true, documents without a value for a field emits
an explicit `null` value which is then added in the composite bucket.
The `missing` option that allows to set an explicit value (instead of `null`) is deprecated in this change and will be removed in a follow up (only in 7.x).
This commit also changes how the big arrays are allocated, instead of reserving
the provided `size` for all sources they are created with a small intial size and they grow
depending on the number of buckets created by the aggregation:
Closes#29380
This commit renames methods in the PersistentTasksService, to
make obvious that the methods send requests in order to change
the state of persistent tasks.
Relates to #29608.
* Make sure all instance variables are final.
* Make generateKey a private static method, instead of protected.
* Rename formatter -> format for consistency.
* Serialize bucket keys as strings as opposed to optional strings.
* Pull the stream serialization logic for buckets into the Bucket class.
Currently AbstractHttpServerTransport is in a netty4 module. This is the
incorrect location. This commit moves it out of netty4 module.
Additionally, it moves unit tests that test AbstractHttpServerTransport
logic to server.
We failed to register "aliases" and "version" into the list of keywords
in the IndexTemplateMetaData; then fail to parse the following index
template.
```
{
"aliases": {"log": {}},
"index_patterns": ["pattern-1"]
}
```
This commit registers that missing keywords.
The stored scripts API today accepts malformed requests instead of throwing an exception.
This PR deprecates accepting malformed put stored script requests (requests not using the official script format).
Relates to #27612
This change replaces some existing try-finally statements that close resources
in their finally block with the slightly shorter and safer try-with-resources
pattern.
This commit removes the method AllocatedPersistentTask.getState() that
exposes the internal state of an AllocatedPersistentTask and replaces
it with a new isCompleted() method. Related to #29608.
Include size of snapshot in snapshot metadata
Adds difference of number of files (and file sizes) between prev and current snapshot. Total number/size reflects total number/size of files in snapshot.
Closes#18543
The BWC version was previously at 7.0, because the 6.x backport had not
yet landed. Now that it has landed, this commit replaces the BWC compat
with the real version, 6.4.0.
Relates #30762
We sign our official plugins yet this is not well-advertised and not at
all consumed during plugin installation. For plugins that are installed
over the intertubes, verifying that the downloaded artifact is signed by
our signing key would establish both integrity and validity of the
downloaded artifact. The chain of trust here is simple: our installable
artifacts (archive and package distributions) so that if a user trusts
our packages via their signatures, and our plugin installer (which would
be executing trusted code) verifies the downloaded plugin, then the user
can trust the downloaded plugin too. This commit adds verification of
official plugins downloaded during installation. We do not add
verification for offline plugin installs; a user can download our
signatures and verify the artifacts themselves.
This commit also needs to solve a few interesting challenges. One of
these is that we want the bouncy castle JARs on the classpath only for
the plugin installer, but not for the runtime
Elasticsearch. Additionally, we want these JARs to not be present for
the JAR hell checks. To address this, we shift these JARs into a
sub-directory of lib (lib/tools/plugin-cli) that is only loaded for the
plugin installer, and in the plugin installer we filter any JARs in this
directory from the JAR hell check.
The writeTo method of VerifyRepositoryResponse incorrectly used its
local version to determine what it was receiving, rather than the
sender's version. This fixes a bug that ocassionally happened when a 6.4
master node sent data to a 7.0 client, causing the number of bytes to be
improperly read. This also unmutes the test.
Closes#30807
Currently nio and netty modules use the CompletableFuture class for
managing listeners. This is unfortunate as that class accepts
Throwable. This commit adds a class CompletableContext that wraps
the CompletableFuture but does not accept Throwable. This allows the
modification of netty and nio logic to no longer handle Throwable.
Treats geohashes as grid cells instead of just points when the
geohashes are used to specify the edges in the geo_bounding_box
query. For example, if a geohash is used to specify the top_left
corner, the top left corner of the geohash cell will be used as the
corner of the bounding box.
Closes#25154
This commit reworks the way our realms perform caching in order to
limit each principal to a single ongoing authentication per realm. In
other words, this means that multiple requests made by the same user
will not trigger more that one authentication attempt at a time if no
entry has been stored in the cache. If an entry is present in our
cache, there is no restriction on the number of concurrent
authentications performed for this user.
This change enables us to limit the load we place on an external system
like an LDAP server and also preserve resources such as CPU on
expensive operations such as BCrypt authentication.
Closes#30355
We now have a remote cluster client exposed which can
talk to a given remote cluster and manages reconnects etc.
This makes code more readable than using the transport layer directly.
Persistent tasks was moved from X-Pack to core in #28455.
However, registration of the named writables and named
X-content was left in X-Pack.
This change moves the registration of the named writables
and named X-content into core. Additionally, the persistent
task actions are no longer registered in the X-Pack client
plugin, as they are already registered in ActionModule.
Today, the `ClusterApplier` and `MasterService` both use the
`ClusterStateTaskListener` interface to notify their callers when asynchronous
activities have completed. However, this is not wholly appropriate: none of the
callers into the `ClusterApplier` care about the `ClusterState` arguments that
they receive. This change introduces a dedicated ClusterApplyListener
interface for callers into the `ClusterApplier`, to distinguish these listeners
from the real `ClusterStateTaskListener`s that are waiting for responses from
the `MasterService`.
This change adds a simple header to the transport client
that is present on the servers thread context that ensures
we can detect if a transport client talks to the server in a
specific request. This change also adds a header for xpack
to detect if the client has xpack installed.
This commit reintroduces 31251c9 and 63a5799. These commits introduced a
memory leak and were reverted. This commit brings those commits back
and fixes the memory leak by removing unnecessary retain method calls.
This reverts commit 31251c9 introduced in #30695.
We suspect this commit is causing the OOME's reported in #30811 and we will use this PR to test this assertion.
Since its introduction in ES 1.4, node fault detection has been using the wrong cluster state version to send
as part of the ping request, by using always the constant -1 (ClusterState.UNKNOWN_VERSION). This can, in an
unfortunate series of events, lead to a situation where a previous stale master can regain its authority and
revert the cluster to an older state.
This commit makes NodesFaultDetection use the correct current cluster state for sending ping requests, avoiding
the situation where a stale master possibly forces a newer master to step down and rejoin the stale one.
With #30672, acking expects *all* nodes to successfully apply the cluster state.
The testElectMasterWithLatestVersion test was checking for an ack while isolating
one node in the test.
Relates to #30672
This commit adds the ability to configure how a docvalue field should be
formatted, so that it would be possible eg. to return a date field
formatted as the number of milliseconds since Epoch.
Closes#27740
The mutate function in UpdateSettingsRequestStreamableTests did not
guarantee that the masterNodeTimeout and timeout values are definitely
changed and occassionally the randomTimeValue() method would select the
sime time value as the original request which caused a failure.
Enables a rolling restart from the OSS distribution to the x-pack based distribution by preventing
x-pack code from installing custom metadata into the cluster state until all nodes are capable of
deserializing this metadata.
When doing a node restart using the test framework, the restarted node does not only use the
settings provided to the original node, but also additional settings provided by plugin extensions,
which does not correspond to the settings that a node would have on a true restart.
The cluster state acking mechanism currently incorrectly acks cluster state updates that have not
successfully been applied on all nodes. In a situation, for example, where some of the nodes
disconnect during publishing, and don't acknowledge receiving the new cluster state, the user-facing
action (e.g. create index request) will still consider this as an ack.
This is related to #29500. We are removing the ability to disable http
pipelining. This PR removes the references to disabling pipelining in
the integration test case.
The VerifyRepositoryResponse class holds a DiscoveryNode[], but the
nodes themselves are not serialized to a REST API consumer. Since we do
not want to put all of a DiscoveryNode over the wire, be it REST or
Transport since its unused, this change introduces a BWC compatible
change in ser/deser of the Response. Anything 6.4 and above will
read/write a NodeView, and anything prior will read/write a
DiscoveryNode. Further changes to 7.0 will be introduced to remove the
BWC shim and only read/write NodeView, and hold a List<NodeView> as the
VerifyRepositoryResponse internal state.
This is code that was leftover from the move to one shard by
default. Here in index metadata we were preserving the default number of
shards settings independently of the area of code where we set this
value on an index that does not explicitly have an number of shards
setting. This took into consideration the es.index.max_number_of_shards
system property, and was used in search requests to set the default
maximum number of concurrent shard requests. We set the default there
based on the default number of shards so that in a one-node case a
search request could concurrently hit all shards on an index with the
defaults. Now that we default to one shard, we expect fewer shards in
clusters and this adjustment of the node count as the max number of
concurrent shard requests is no longer needed. This commit then changes
the default number of shards settings to be consistent with the value
used when an index is created, and removes the now unneeded adjustment
in search requests.
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.
Closes#30741Closes#30577
This is related to #29500 and #28898. This commit removes the abilitiy
to disable http pipelining. After this commit, any elasticsearch node
will support pipelined requests from a client. Additionally, it extracts
some of the http pipelining work to the server module. This extracted
work is used to implement pipelining for the nio plugin.
We added this limit because we occasionally saw cases where most of the memory
usage of the cache was spent on the keys (ie. queries) rather than the values,
which caused the cache to vastly underestimate its memory usage. In recent
releases, we disabled caching on heavy `terms` queries, which were the main
source of the problem, so putting more entries in the cache should be safer.
The test has an issue that exhibits only super rarely. The test sets the publish
timeout to 0, then proceeds to block cluster state processing on a data node,
then deletes an index and recreates it, and finally removes the cluster state
processing block. Finally, it calls ensureGreen, which might now return before
the data node has fully applied the cluster state that removed and readded the
shard, due to the publish timeout of 0. This commit waits for the cluster state
to be fully processed on the data node before doing the search.
Closes#30718
This change makes sure that an empty completion input does not throw an IAE when indexing.
Instead the input is ignored and the completion field is added in the list of ignored fields
for the document.
Closes#23121
This is related to #27260. The elasticsearch-nio jar is supposed to be
a library opposed to a framework. Currently it internally logs certain
exceptions. This commit modifies it to not rely on logging. Instead
exception handlers are passed by the applications that use the jar.
This commit adds Delete Repository, the associated docs and tests for
the high level REST API client. It also cleans up a seemingly innocuous
line in the RestDeleteRepositoryAction and some naming in SnapshotIT.
Relates #27205
The copy_settings parameter will be removed in Elasticsearch 8.0.0. This
commit adds an assertion message that to clean up this code when master
is bumped to 8.0.0.
Added dedicated script contexts for:
* script function score
* script sorting
* terms_set query
Scripts for these contexts will either have a specific return value or
use scoring and therefor in the future will need their own scripting classes.
Relates to #30511
The getDate() and getDates() existed prior to 5.x on long fields in
scripting. In 5.x, a new Date type for ScriptDocValues was added. The
getDate() and getDates() methods were left on long fields and added to date
fields to ease the transition. This commit removes those methods for
7.0.
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
I still do not like == false. However, I am so use to reading it that
today I read this line of code and could not understand how it could
possibly be doing the right thing. It was only when I finally noticed
the ! that the code made sense. This commit changes this code to be in
our style of == false. I still do not like == false.
Get Settings API changes have now been backported to version 6.4, and
therefore the latest version must send and expect the extra fields when
communicating with 6.4+ code.
Relates #29229#30494
Currently in a rescore request if window_size is smaller than
the top N documents returned (N=size), explanation of scores could be incorrect
for documents that were a part of topN and not part of rescoring.
This PR corrects this, but saving in RescoreContext docIDs of documents
for which rescoring was applied, and adding rescoring explanation
only for these docIDs.
Closes#28725
The camel case name `nGram` should be removed in favour of `ngram` and
similar for `edgeNGram` and `edge_ngram`. Before removal, we need to
deprecate the camel case names first. This change adds deprecation
warnings for indices with versions 6.4.0 and higher and logs deprecation
warnings.
Since #30143, the Cluster State API should always returns the current
cluster_uuid in the response body, regardless of the metrics filters.
This is not exactly true as it is returned only if metadata metrics and
no specific indices are requested.
This commit fixes the behavior to always return the cluster_uuid and
add new test.
This test failed but the cause is not obvious. This commit adds more
debug logging traces so that if it reproduces we could gather more
information.
Related #30577
Date histograms on non-fixed timezones such as `Europe/Paris` proved much slower
than histograms on fixed timezones in #28727. This change mitigates the issue by
using a fixed time zone instead when shard data doesn't cross a transition so
that all timestamps share the same fixed offset. This should be a common case
with daily indices.
NOTE: Rewriting the aggregation doesn't work since the timezone is then also
used on the coordinating node to create empty buckets, which might be out of the
range of data that exists on the shard.
NOTE: In order to be able to get a shard context in the tests, I reused code
from the base query test case by creating a new parent test case for both
queries and aggregations: `AbstractBuilderTestCase`.
Mitigates #28727
This pipeline aggregation gives the user the ability to script functions that "move" across a window
of data, instead of single data points. It is the scripted version of MovingAvg pipeline agg.
Through custom script contexts, we expose a number of convenience methods:
- MovingFunctions.max()
- MovingFunctions.min()
- MovingFunctions.sum()
- MovingFunctions.unweightedAvg()
- MovingFunctions.linearWeightedAvg()
- MovingFunctions.ewma()
- MovingFunctions.holt()
- MovingFunctions.holtWinters()
- MovingFunctions.stdDev()
The user can also define any arbitrary logic via their own scripting, or combine with the above methods.
The TemplateUpgradeService is a system service that allows for plugins
to register templates that need to be upgraded. These template upgrades
should always happen in a system context as they are not a user
initiated action. For security integrations, the lack of running this
in a system context could lead to unexpected failures. The changes in
this commit set an empty system context for the execution of the
template upgrades performed by this service.
Relates #30603
When processing a top-level sibling pipeline, we destructively sublist
the path by assigning back onto the same variable. But if aggs are
specified such:
A. Multi-bucket agg in the first entry of our internal list
B. Regular agg as the immediate child of the multi-bucket in A
C. Regular agg with the same name as B at the top level, listed as the
second entry in our internal list
D. Finally, a pipeline agg with the path down to B
We'll get class cast exception. The first agg will sublist the path
from [A,B] to [B], and then when we loop around to check agg C,
the sublisted path [B] matches the name of C and it fails.
The fix is simple: we just need to store the sublist in a new object
so that the old path remains valid for the rest of the aggs in the loop
Closes#30608
* Fixes IndiceOptionsTests to serialise correctly
Previous to this change `IndicesOptionsTests.testSerialisation()` would
select a complete random version for both the `StreamOutput` and the
`StreamInput`. This meant that the output could be selected as 7.0+
while the input was selected as <7.0 causing the stream to be written
in the new format and read in teh old format (or vica versa). This
change splits the two cases into different test methods ensuring that
the Streams are at least on compatibile versions even if they are on
different versions.
* Use same random version for input and output streams
server/src/test/java/org/elasticsearch/action/support/IndicesOptionsTest
s.java
This change adds a `listTasks` method to the high level java
ClusterClient which allows listing running tasks through the
task management API.
Related to #27205
Allows the setting to be specified using proper array syntax, for example:
"cluster.routing.allocation.awareness.attributes": [ "foo", "bar", "baz" ]
Closes#30617
This commit adds Create Repository, the associated docs and tests
for the high level REST API client. A few small changes to the
PutRepository Request and Response went into the commit as well.
This commit is related to #28898. It adds an nio driven http server
transport. Currently it only supports basic http features. Cors,
pipeling, and read timeouts will need to be added in future PRs.
* Refactor IndicesOptions to not be byte-based
This refactors IndicesOptions to be enum/enummap based rather than using a byte
as a bitmap for each of the options. This is necessary because we'd like to add
additional options, but we ran out of bits.
Backwards compatibility is kept for earlier versions so the option serialization
does not change the options.
Relates sort of to #30188
When we split/shrink an index we open several IndexWriter instances
causeing file-deletes to be pending on windows. This subsequently fails
when we open an IW to bootstrap the index history due to pending deletes.
This change sidesteps the check since we know our history goes forward
in terms of files and segments.
Closes#30416
Ryan Ernst
Jim Ferenczi
Tanguy Leroux
Nhat Nguyen
Michael Basnight
Jim Ferenczi
Christoph Büscher
Luca Cavanna
olcbean
Yannick Welsch
Alan Woodward
Julie Tibshirani
Tim Brooks
Martijn van Groningen
Sohaib Iftikhar
Albert Zaharovits
Vladimir Dolzhenko
Jason Tedor
Igor Motov
Jay Modi
Simon Willnauer
David Roberts
David Turner
Colin Goodheart-Smithe
Adrien Grand
Lee Hinman
Boaz Leskes
tomcallahan
Mayya Sharipova
Zachary Tong
Ke Li
Zachary Tong
Van0SS
Lee Hinman