This commit allows us to use different TranslogRecoveryRunner when
recovering an engine from its local translog. This change is a
prerequisite for the commit-based rollback PR.
Relates #32867
The main benefit of the upgrade for users is the search optimization for top scored documents when the total hit count is not needed. However this optimization is not activated in this change, there is another issue opened to discuss how it should be integrated smoothly.
Some comments about the change:
* Tests that can produce negative scores have been adapted but we need to forbid them completely: #33309Closes#32899
When index sorting is enabled, toXContent tried to serialize an
SortField object, resulting in an exception, when using the _segments
endpoint.
Relates #29120
With this commit we use the classic parent circuit breaker which does
not account for real memory usage. In those tests we want to have
reproducible results and hence it makes sense to disable the real memory
circuit breaker there.
Auto Following Patterns is a cross cluster replication feature that
keeps track whether in the leader cluster indices are being created with
names that match with a specific pattern and if so automatically let
the follower cluster follow these newly created indices.
This change adds an `AutoFollowCoordinator` component that is only active
on the elected master node. Periodically this component checks the
the cluster state of remote clusters if there new leader indices that
match with configured auto follow patterns that have been defined in
`AutoFollowMetadata` custom metadata.
This change also adds two new APIs to manage auto follow patterns. A put
auto follow pattern api:
```
PUT /_ccr/_autofollow/{{remote_cluster}}
{
"leader_index_pattern": ["logs-*", ...],
"follow_index_pattern": "{{leader_index}}-copy",
"max_concurrent_read_batches": 2
... // other optional parameters
}
```
and delete auto follow pattern api:
```
DELETE /_ccr/_autofollow/{{remote_cluster_alias}}
```
The auto follow patterns are directly tied to the remote cluster aliases
configured in the follow cluster.
Relates to #33007
Co-authored-by: Jason Tedor jason@tedor.me
With features like CCR building on the CCS infrastructure, the settings
prefix search.remote makes less sense as the namespace for these remote
cluster settings than does a more general namespace like
cluster.remote. This commit replaces these settings with cluster.remote
with a fallback to the deprecated settings search.remote.
There are two races in the testUpdateAndReadChangesConcurrently if the
following engines are created in the worker threads. We fixed the
translog issue in #33352, but there is still another race with
createStore.
This commit ensures that we create all engines in the main thread.
Relates #33352Closes#33344
Historically we have had a ESLoggingHandler in the netty module that
logs low-level connection operations. This class just extends the netty
logging handler with some (broken) message deserialization. This commit
fixes this message serialization and moves the class to server.
This new logger logs inbound and outbound messages. Eventually, we
should move other event logging to this class (connect, close, flush).
That way we will have consistent logging regards of which transport is
loaded.
Resolves#27306 on master. Older branches will need a different fix.
This commit is related to #32517. It allows an "server_name"
attribute on a DiscoveryNode to be propagated to the server using
the TLS SNI extentsion. This functionality is only implemented for
the netty security transport.
This change adds an expert index setting called `index.merge.policy.deletes_pct_allowed`.
It controls the maximum percentage of deleted documents that is tolerated in the index.
Lower values make the index more space efficient at the expense of increased CPU and I/O activity.
Values must be between `20` and `50`. Default value is `33`.
Drops and unused logging constructor, simplifies a rarely used one, and
removes `Settings` from a third. There is now only a single logging ctor
that takes `Settings` and we'll remove that one in a follow up change.
The maximum number of fields per index is limited to 1000 by default by the
`index.mapping.total_fields.limit` setting to prevent accidental mapping
explosions due to too many fields. Currently all metadata fields also count
towards this limit, which can lead to some confusion when using lower limits.
It is not obvious for users that they cannot actually add as many fields as
are specified by the limit in this case.
This change takes the number of metadata fields out of the field count that we
check against the field limit. It also adds tests that check that we can add
fields up to the specified limit, but throw an exception for any additional field added.
Closes#24096
Deprecating a some setting specializations (e.g., list settings) does
not cause deprecation warning headers and deprecation log messages to
appear. This is due to a missed check for deprecation. This commit fixes
this for all setting specializations, and ensures that this can not be
missed again.
Adds a place for users to store cluster-wide data they wish to associate
with the cluster via the Cluster Settings API. This is strictly for
user-defined data, Elasticsearch makes no other other use of these
settings.
Now that types are unique per mapping we can retrieve the document mapper
without referencing the type. This fixes an NPE when stored fields are disabled.
For 6x we'll need a different fix since mappings can still have multiple types.
Relates #32941
This commit introduces the formal notion of a private setting. This
enables us to register some settings that we had previously not
registered as fully-fledged settings to avoid them being exposed via
APIs such as the create index API. For example, we had hacks in the
codebase to allow index.version.created to be passed around inside of
settings objects, but was not registered as a setting so that if a user
tried to use the setting on any API then they would get an
exception. This prevented users from setting index.version.created on
index creation, or updating it via the index settings API. By
introducing private settings, we can continue to reject these attempts,
yet now we can represent these settings as actual settings. In this
change, we register index.version.created as an actual setting. We do
not cutover all settings that we had been treating as private in this
pull request, it is already quite large due to moving some tests around
to account for the fact that some tests need to be able to set the
index.version.created. This can be done in a follow-up change.
* TESTS: Fix Race Condition in Temp Path Creation
* Calling `createTempDir` concurrently here in
the `Follower`s causes collisions at times
which lead to `createEngine` throwing because
of unexpected files in the newly created temp
dir
* Fixed by creating all temp dirs in the main test thread
* closes#33344
We can have multiple documents in Lucene with the same seq_no for
parent-child documents (or without rollback). In this case, the usage
"lastSeenSeqNo + 1" is an off-by-one error as it may miss some
documents. This error merely affects the `skippedOperations` contract.
See: https://github.com/elastic/elasticsearch/pull/33222#discussion_r213842257Closes#33318
The existing implemention could not deal with negative numbers as well
as +- 999 milliseconds around the epoch.
This commit uses Instant.ofEpochMilli() and parses the input to
a number instead of using a date formatter.
This commit adds the support to early terminate the collection of a leaf
in the aggregation framework. This change introduces a MultiBucketCollector which
handles CollectionTerminatedException exactly like the Lucene MultiCollector.
Any aggregator can now throw a CollectionTerminatedException without stopping
the collection of a sibling aggregator. This is useful for aggregators that
can infer their result without visiting all documents (e.g.: a min/max aggregation on a match_all query).
Drops `Settings` from some logging ctors now that they are no longer
needed. This should allow us to stop passing `Settings` around to quite
as many places.
* master:
Mute test watcher usage stats output
[Rollup] Fix FullClusterRestart test
Adjust soft-deletes version after backport into 6.5
completely drop `index.shard.check_on_startup: fix` for 7.0 (#33194)
Fix AwaitsFix issue number
Mute SmokeTestWatcherWithSecurityIT testsi
drop `index.shard.check_on_startup: fix` (#32279)
tracked at
[DOCS] Moves ml folder from x-pack/docs to docs (#33248)
[DOCS] Move rollup APIs to docs (#31450)
[DOCS] Rename X-Pack Commands section (#33005)
TEST: Disable soft-deletes in ParentChildTestCase
Fixes SecurityIntegTestCase so it always adds at least one alias (#33296)
Fix pom for build-tools (#33300)
Lazy evaluate java9home (#33301)
SQL: test coverage for JdbcResultSet (#32813)
Work around to be able to generate eclipse projects (#33295)
Highlight that index_phrases only works if no slop is used (#33303)
Different handling for security specific errors in the CLI. Fix for https://github.com/elastic/elasticsearch/issues/33230 (#33255)
[ML] Refactor delimited file structure detection (#33233)
SQL: Support multi-index format as table identifier (#33278)
MINOR: Remove Dead Code from PathTrie (#33280)
Enable forbiddenapis server java9 (#33245)
This PR integrates Lucene soft-deletes(LUCENE-8200) into Elasticsearch.
Highlight works in this PR include:
- Replace hard-deletes by soft-deletes in InternalEngine
- Use _recovery_source if _source is disabled or modified (#31106)
- Soft-deletes retention policy based on the global checkpoint (#30335)
- Read operation history from Lucene instead of translog (#30120)
- Use Lucene history in peer-recovery (#30522)
Relates #30086Closes#29530
---
These works have been done by the whole team; however, these individuals
(lexical order) have significant contribution in coding and reviewing:
Co-authored-by: Adrien Grand <jpountz@gmail.com>
Co-authored-by: Boaz Leskes <b.leskes@gmail.com>
Co-authored-by: Jason Tedor <jason@tedor.me>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co>
Co-authored-by: Simon Willnauer <simonw@apache.org>
This PR integrates Lucene soft-deletes(LUCENE-8200) into Elasticsearch.
Highlight works in this PR include:
- Replace hard-deletes by soft-deletes in InternalEngine
- Use _recovery_source if _source is disabled or modified (#31106)
- Soft-deletes retention policy based on the global checkpoint (#30335)
- Read operation history from Lucene instead of translog (#30120)
- Use Lucene history in peer-recovery (#30522)
Relates #30086Closes#29530
---
These works have been done by the whole team; however, these individuals
(lexical order) have significant contribution in coding and reviewing:
Co-authored-by: Adrien Grand jpountz@gmail.com
Co-authored-by: Boaz Leskes b.leskes@gmail.com
Co-authored-by: Jason Tedor jason@tedor.me
Co-authored-by: Martijn van Groningen martijn.v.groningen@gmail.com
Co-authored-by: Nhat Nguyen nhat.nguyen@elastic.co
Co-authored-by: Simon Willnauer simonw@apache.org
This PR removes the deprecated `Custom` class in `IndexMetaData`, in favor
of a `Map<String, DiffableStringMap>` that is used to store custom index
metadata. As part of this, there is now no way to set this metadata in a
template or create index request (since it's only set by plugins, or dedicated
REST endpoints).
The `Map<String, DiffableStringMap>` is intended to be a namespaced `Map<String,
String>` (`DiffableStringMap` implements `Map<String, String>`, so the signature
is more like `Map<String, Map<String, String>>`). This is so we can do things
like:
``` java
Map<String, String> ccrMeta = indexMetaData.getCustom("ccr");
```
And then have complete control over the metadata. This also means any
plugin/feature that uses this has to manage its own BWC, as the map is just
serialized as a map. It also means that if metadata is put in the map that isn't
used (for instance, if a plugin were removed), it causes no failures the way
an unregistered `Setting` would.
The reason I use a custom `DiffableStringMap` here rather than a plain
`Map<String, String>` is so the map can be diffed with previous cluster state
updates for serialization.
Supersedes #32683
We cut over to `index.resize.source.*` but still have these constants
being public in `IndexMetaData`. Those Settings and constants are not needed
in 7.x while we still need to keep the keys known to private settings since
they might be part of the index settings of old indices. We can remove that
in 8.0. Yet, we should remove the settings to make sure they are not used again.
Fix serialization of empty field capabilities response
When no response are required (no indices match the requested patterns) the
empty response throws an NPE in the transport serialization (writeTo).
If an exclude or an include clause removes an entry to a nested field in the original source at query time,
the creation of nested hits fails with an NPE. This change fixes this exception and replaces the nested document
source with an empty map.
Closes#33163Closes#33170
Today we add a NoOp to Lucene and translog if we fail to process an
indexing operation. However, we are only adding NoOps to translog for
delete operations. In order to have a complete history in Lucene, we
should add NoOps of failed delete operations to both Lucene and translog.
Relates #29530
Today we support a static list of seed hosts in core Elasticsearch, and allow a
dynamic list of seed hosts to be provided via a file using the `discovery-file`
plugin. In fact the ability to provide a dynamic list of seed hosts is
increasingly useful, so this change moves this functionality to core
Elasticsearch to avoid the need for a plugin.
Furthermore, in order to start up nodes in integration tests we currently
assign a known port to each node before startup, which unfortunately sometimes
fails if another process grabs the selected port in the meantime. By moving the
`discovery-file` functionality into the core product we can use it to avoid
this race.
This change also moves the expected path to the file from
`$ES_PATH_CONF/discovery-file/unicast_hosts.txt` to
`$ES_PATH_CONF/unicast_hosts.txt`. An example of this file is not included in
distributions.
For BWC purposes the plugin still exists, but does nothing more than create the
example file in the old location, and issue a warning when it is used. We also
continue to support the old location for the file, but warn about its
deprecation.
Relates #29244Closes#33030
Some settings have filters applied to them and we use this in logs and
the get nodes info API. For consistency, we should apply this in the get
cluster settings API too.
* master:
Painless: Add Bindings (#33042)
Update version after client credentials backport
Fix forbidden apis on FIPS (#33202)
Remote 6.x transport BWC Layer for `_shrink` (#33236)
Test fix - Graph HLRC tests needed another field adding to randomisation exception list
HLRC: Add ML Get Records API (#33085)
[ML] Fix character set finder bug with unencodable charsets (#33234)
TESTS: Fix overly long lines (#33240)
Test fix - Graph HLRC test was missing field name to be excluded from randomisation logic
Remove unsupported group_shard_failures parameter (#33208)
Update BucketUtils#suggestShardSideQueueSize signature (#33210)
Parse PEM Key files leniantly (#33173)
INGEST: Add Pipeline Processor (#32473)
Core: Add java time xcontent serializers (#33120)
Consider multi release jars when running third party audit (#33206)
Update MSI documentation (#31950)
HLRC: create base timed request class (#33216)
[DOCS] Fixes command page titles
HLRC: Move ML protocol classes into client ml package (#33203)
Scroll queries asking for rescore are considered invalid (#32918)
Painless: Fix Semicolon Regression (#33212)
ingest: minor - update test to include dissect (#33211)
Switch remaining LLREST usage to new style Requests (#33171)
HLREST: add reindex API (#32679)
The shrink action was renamed to `_resize` with the addition
or split. This bwc layer is unnecessary on 7.x since 6.latest will
always use the resize action.
We have had support for the `group_shard_failures` parameter in our code for a while, since we introduced failures grouping. When we introduced validation of parameters at REST, we seem to have forgotten to expose such parameter. Given that the parameter is effectively not supported for many months now, that no user has complained about that and that grouping is the expected behaviour, this commit removes support for the parameter.
`BucketUtils#suggestShardSideQueueSize` used to calculate the shard_size based on the number of shards. It returns now a different value only based on whether we are querying a single shard or multiple shards. This commit replaces the numberOfShards argument with a boolean that tells whether we are querying a single shard or not.
Exclude classes meant for newer versions than what we are auditing against, those classes won't be found. There's no reason to exclude JDK classes from newer versions, with this PR, we will not extract them in the first place.
* master:
[Rollup] Better error message when trying to set non-rollup index (#32965)
HLRC: Use Optional in validation logic (#33104)
Remove unused User class from protocol (#33137)
ingest: Introduce the dissect processor (#32884)
[Docs] Add link to es-kotlin-wrapper-client (#32618)
[Docs] Remove repeating words (#33087)
Minor spelling and grammar fix (#32931)
Remove support for deprecated params._agg/_aggs for scripted metric aggregations (#32979)
Watcher: Simplify finding next date in cron schedule (#33015)
Run Third party audit with forbidden APIs CLI (part3/3) (#33052)
Fix plugin build test on Windows (#33078)
HLRC+MINOR: Remove Unused Private Method (#33165)
Remove old unused test script files (#32970)
Build analysis-icu client JAR (#33184)
Ensure to generate identical NoOp for the same failure (#33141)
ShardSearchFailure#readFrom to set index and shardId (#33161)
Today a file-based recovery will replay all existing translog operations
from the primary on a replica so that that replica can have a full
history in translog as the primary. However, with soft-deletes enabled,
we should not do it because:
1. All operations before the local checkpoint of the safe commit exist in
the commit already.
2. The number of operations before the local checkpoint may be considerable
and requires a significant amount of time to replay on a replica.
Relates #30522
Relates #29530
This commit makes primary-replica resyncer use Lucene as the source of
history operation instead of translog if soft-deletes is enabled. With
this change, we no longer expose translog snapshot directly in IndexShard.
Relates #29530
The new implementation is functional equivalent with the old, ant based one.
It parses task standard error to get the missing classes and violations in the same way.
I considered re-using ForbiddenApisCliTask but Gradle makes it hard to build inheritance with tasks that have task actions , since the order of the task actions can't be controlled.
This inheritance isn't dully desired either as the third party audit task is much more opinionated and we don't want to expose some of the configuration.
We could probably extract a common base class without any task actions, but probably more trouble than it's worth.
Closes#31715
We generate slightly different NoOps in InternalEngine and
TransportShardBulkAction for the same failure.
1. InternalEngine uses Exception#getFailure to generate a message
without the class name: newOp [NoOp{seqNo=1, primaryTerm=1,
reason='Contexts are mandatory in context enabled completion field
[suggest_context]'}].
2. TransportShardBulkAction uses Exception#toString to generate a
message with the class name: NoOp{seqNo=1, primaryTerm=1,
reason='java.lang.IllegalArgumentException: Contexts are mandatory in
context enabled completion field [suggest_context]'}.
If a write operation fails while a replica is recovering, that replica
will possibly receive two different NoOps: one from recovery and one
from replication. These two different NoOps will trip
TranslogWriter#assertNoSeqNumberConflict assertion.
This commit ensures that we generate the same Noop for the same failure.
Closes#32986
As part of recent changes made to `ShardOperationFailedException` we introduced `index` and `shardId` members to the base class, but the subclasses are entirely responsible for the serialization of such fields. In the case of `ShardSearchFailure`, we have an additional `SearchShardTarget` instance member which also holds the index and the shardId, hence they get serialized as part of `SearchShardTarget` itself. When de-serializing a `ShardSearchFailure` though, we need to remember to also set the parent class `index` and `shardId` fields otherwise they get lost
Relates to #32640
* master:
Adjust BWC version on mapping version
Token API supports the client_credentials grant (#33106)
Build: forked compiler max memory matches jvmArgs (#33138)
Introduce mapping version to index metadata (#33147)
SQL: Enable aggregations to create a separate bucket for missing values (#32832)
Fix grammar in contributing docs
SECURITY: Fix Compile Error in ReservedRealmTests (#33166)
APM server monitoring (#32515)
Support only string `format` in date, root object & date range (#28117)
[Rollup] Move toBuilders() methods out of rollup config objects (#32585)
Fix forbiddenapis on java 11 (#33116)
Apply publishing to genreate pom (#33094)
Have circuit breaker succeed on unknown mem usage
Do not lose default mapper on metadata updates (#33153)
Fix a mappings update test (#33146)
Reload Secure Settings REST specs & docs (#32990)
Refactor CachingUsernamePassword realm (#32646)
The introduction of mapping version on index metadata has been
backported to 6.x. This commit adjusts the BWC version around mapping
version to account for this backport.
This commit introduces mapping version to index metadata. This value is
monotonically increasing and is updated on mapping updates. This will be
useful in cross-cluster replication so that we can request mapping
updates from the leader only when there is a mapping update as opposed
to the strategy we employ today which is to request a mapping update any
time there is an index metadata update. As index metadata updates can
occur for many reasons other than mapping updates, this leads to some
unnecessary requests and work in cross-cluster replication.
With this commit we implement a workaround for
https://bugs.openjdk.java.net/browse/JDK-8207200 which is a race
condition in the JVM that results in `IllegalArgumentException` to be
thrown in rare cases when we determine memory usage via `MemoryMXBean`.
As we do not want to fail requests in those cases we always return zero
memory usage.
Relates #31767
Relates #33125
When applying index metadata updates we run through the mappings
updating them if needed. Today if there is not an update to the default
mapper, we can lose the default mapping. This means that, for example,
if we apply a settings update to an index we will lose the default
mapper. This happens because we were not guarding updating the default
mapping with a check that the default mapping was updated in the
metadata update. When there is no update in the metadata update, we need
to continue to preserve the previous default mapping. This commit
achieves this by moving the updating of the default mapping under the
same guard that we use for updating the default mapping source. We add a
test that fails before putting the update under a guard and now passes
after moving the update under the guard.
This commit fixes a mappings update test. The test is broken in the
sense that it passes, but for the wrong reason. The test here is testing
that if we make a mapping update but do not commit that mapping update
then the mapper service still maintains the previous document
mapper. This was not the case long, long ago when a mapping update would
update the in-memory state before the cluster state update was
committed. This test was passing, but it was passing because the mapping
update was never even updated. It was never even updated because it was
encountering a null pointer exception. Of course the in-memory state is
not going to be updated in that case, we are simply going to end up with
a failed cluster state update. Fixing that leads to another issue which
is that the mapping source does not even parse so again we would, of
course, end up with the in-memory state not being modified. We fix these
issues, assert that the result cluster state task completed
successfully, and finally that the in-memory state was not updated since
we never committed the resulting cluster state.
* master:
Add proxy support to RemoteClusterConnection (#33062)
TEST: Skip assertSeqNos for closed shards (#33130)
TEST: resync operation on replica should acquire shard permit (#33103)
Switch remaining x-pack tests to new style Requests (#33108)
Switch remaining tests to new style Requests (#33109)
Switch remaining ml tests to new style Requests (#33107)
Build: Line up IDE detection logic
Security index expands to a single replica (#33131)
HLRC: request/response homogeneity and JavaDoc improvements (#33133)
Checkstyle!
[Test] Fix sporadic failure in MembershipActionTests
Revert "Do NOT allow termvectors on nested fields (#32728)"
[Rollup] Move toAggCap() methods out of rollup config objects (#32583)
Fix race condition in scheduler engine test
This adds support for connecting to a remote cluster through
a tcp proxy. A remote cluster can configured with an additional
`search.remote.$clustername.proxy` setting. This proxy will be used
to connect to remote nodes for every node connection established.
We still try to sniff the remote clsuter and connect to nodes directly
through the proxy which has to support some kind of routing to these nodes.
Yet, this routing mechanism requires the handshake request to include some
kind of information where to route to which is not yet implemented. The effort
to use the hostname and an optional node attribute for routing is tracked
in #32517Closes#31840
If a shard was closed, we return null for SeqNoStats. Therefore the
assertion assertSeqNos will hit NPE when it verifies a closed shard.
This commit skips closed shards in assertSeqNos and enables this
assertion in AbstractDisruptionTestCase.
* es/master: (62 commits)
[DOCS] Add docs for Application Privileges (#32635)
Add versions 5.6.12 and 6.4.1
Do NOT allow termvectors on nested fields (#32728)
[Rollup] Return empty response when aggs are missing (#32796)
[TEST] Add some ACL yaml tests for Rollup (#33035)
Move non duplicated actions back into xpack core (#32952)
Test fix - GraphExploreResponseTests should not randomise array elements Closes#33086
Use `addIfAbsent` instead of checking if an element is contained
TESTS: Fix Random Fail in MockTcpTransportTests (#33061)
HLRC: Fix Compile Error From Missing Throws (#33083)
[DOCS] Remove reload password from docs cf. #32889
HLRC: Add ML Get Buckets API (#33056)
Watcher: Improve error messages for CronEvalTool (#32800)
Search: Support of wildcard on docvalue_fields (#32980)
Change query field expansion (#33020)
INGEST: Cleanup Redundant Put Method (#33034)
SQL: skip uppercasing/lowercasing function tests for AZ locales as well (#32910)
Fix the default pom file name (#33063)
Switch ml basic tests to new style Requests (#32483)
Switch some watcher tests to new style Requests (#33044)
...
* Search: Support of wildcard on docvalue_fields
For consistency with stored_fields, docvalue_fields should support the use of wildcards.
Documentation of doc values fields is updated accordingly.
See also: #26390Closes#26299
This commit changes the query field expansion for query parsers
to not rely on an hardcoded list of field types. Instead we rely on
the type of exception that is thrown by MappedFieldType#termQuery to
include/exclude an expanded field.
Supersedes #31655Closes#31798
We used to set `maxScore` to `0` within `TopDocs` in situations where there is really no score as the size was set to `0` and scores were not even tracked. In such scenarios, `Float.Nan` is more appropriate, which gets converted to `max_score: null` on the REST layer. That's also more consistent with lucene which set `maxScore` to `Float.Nan` when merging empty `TopDocs` (see `TopDocs#merge`).
In our Netty layer we have had to take extra precautions against Netty
catching throwables which prevents them from reaching the uncaught
exception handler. This code has taken on additional uses in NIO layer
and now in the scheduler engine because there are other components in
stack traces that could catch throwables and suppress them from reaching
the uncaught exception handler. This commit is a simple cleanup of the
iterative evolution of this code to refactor all uses into a single
method in ExceptionsHelper.
Today we can only have non-affix settings updated and consumed _together_.
Yet, there are use-cases where two affix settings depend on each other which
makes using the hard without consuming updates together. Unfortunately, there is
not straight forward way to have N settings updated together in a type-safe way
having 2 still serves a large portion of use-cases.
This change allows an engine to recover from its local translog up to
the given seqno. The extended API can be used in these use cases:
When a replica starts following a new primary, it resets its index to
the safe commit, then replays its local translog up to the current
global checkpoint (see #32867).
When a replica starts a peer-recovery, it can initialize the
start_sequence_number to the persisted global checkpoint instead of the
local checkpoint of the safe commit. A replica will then replay its
local translog up to that global checkpoint before accepting remote
translog from the primary. This change will increase the chance of
operation-based recovery. I will make this in a follow-up.
Relates #32867
Today `_msearch` doesn't allow modifying the `max_concurrent_shard_requests`
per sub search request. This change adds support for setting this parameter on
all sub-search requests in an `_msearch`.
Relates to #31877
The maximum map count boostrap check can be a hindrance to users that do
not own the underlying platform on which they are executing
Elasticsearch. This is because addressing it requires tuning the kernel
and a platform provider might now allow this, especially on shared
infrastructure. However, this bootstrap check is not needed if mmapfs is
not in use. Today we do not have a way for the user to communicate that
they are not going to use mmapfs. This commit therefore adds a setting
that enables the user to disallow mmapfs. When mmapfs is disallowed, the
maximum map count bootstrap check is not enforced. Additionally, we
fallback to a different default index store and prevent the explicit use
of mmapfs for an index.
This change introduces a dedicated ConnectionManager for every RemoteClusterConnection
such that there is not state shared with the TransportService internal ConnectionManager.
All connections to a remote cluster are isolated from the TransportService but still uses
the TransportService and it's internal properties like the Transport, tracing and internal
listener actions on disconnects etc.
This allows a remote cluster connection to have a different lifecycle than a local cluster connection,
also local discovery code doesn't get notified if there is a disconnect on from a remote cluster and
each connection can use it's own dedicated connection profile which allows to have a reduced set of
connections per cluster without conflicting with the local cluster.
Closes#31835
* INGEST: Move all Pipeline State into IngestService
* Moves all pipeline state into the ingest service
* Retains the existing pipeline store and pipeline execution service as inner classes to make the review easier, they should be flattened out in the next step
* All tests for these classes were copied (and adapted) to the ingest service tests
* This is a refactoring step to enable a clean implementation of a pipeline processor (See #32473)
There are two problems with the scheduler engine today. Both relate to
listeners that throw.
The first problem is that any triggered listener that throws a plain old
exception will cause no additional listeners to be triggered for the
event, and will also cause the scheduler to never be invoked again. This
leads to lost events and is bad.
The second problem is that any triggered listener that throws an error
of the fatal kind will not lead to that error because caught by the
uncaught exception handler. This is because the triggered listener is
executed as a future task under a scheduled thread pool executor. A
throwable there goes caught by the JDK framework and set as the outcome
on the future task. Since we never inspect these tasks for their
outcomes, nor is there a good place to do this, we have to handle these
errors ourselves. To do this, we catch them and dispatch them to the
uncaught exception handler via a forked thread. This is similar to our
handling in Netty.
* master:
Generalize remote license checker (#32971)
Trim translog when safe commit advanced (#32967)
Fix an inaccuracy in the dynamic templates documentation. (#32890)
Logging: Use settings when building daemon threads (#32751)
All Translog inner closes should happen after tragedy exception is set (#32674)
HLREST: AwaitsFix ML Test
Pass DiscoveryNode to initiateChannel (#32958)
Add mzn and dz to unsupported locales (#32957)
Use settings from the context in BootstrapChecks (#32908)
Update docs for node specifications (#30468)
HLRC: Forbid all Elasticsearch logging infra (#32784)
Only configure publishing if it's applied externally (#32351)
Fixes libs:dissect when in eclipse
Protect ScriptedMetricIT test cases against failures on 0-doc shards (#32959) (#32968)
[Kerberos] Add documentation for Kerberos realm (#32662)
Watcher: Properly find next valid date in cron expressions (#32734)
Fix some small issues in the getting started docs (#30346)
Set forbidden APIs target compatibility to compiler java version (#32935)
Move connection listener to ConnectionManager (#32956)
Since #28140 when the global checkpoint is advanced, we try to move the
safe commit forward, and clean up old index commits if possible. However,
we forget to trim unreferenced translog.
This change makes sure that we prune both old translog and index commits
when the safe commit advanced.
Relates #28140Closes#32089
Subclasses of `EsIntegTestCase` run multiple Elasticsearch nodes in the
same JVM and when we log we look at the name of the thread to figure out
the node name. This makes sure that all calls to `daemonThreadFactory`
include the node name.
Closes#32574
I'd like to follow this up with more drastic changes that make it
impossible to do this incorrectly but that change is much larger than
this and I'd like to get these log lines fixed up sooner rather than
later.
All Translog inner closes should happen after tragedy exception is set (#32674)
We faced with the nasty race condition. See #32526
InternalEngine.failOnTragic method has thrown AssertionError.
If you carefully look at if branches in this method, you will spot that its only possible, if either Lucene IndexWriterhas closed from inside or Translog, has closed from inside, but tragedy exception is not set.
For now, let us concentrate on the Translog class.
We found out that there are two methods in Translog - namely rollGeneration and trimOperations that are closing Translog in case of Exception without tragedy exception being set.
This commit fixes these 2 methods. To fix it, we pull tragedyException from TranslogWriter up-to Translog class, because in these 2 methods IndexWriter could be innocent, but still Translog needs to be closed. Also, tragedyException is wrapped with TragicExceptionHolder to reuse CAS/addSuppresed functionality in Translog and TranslogWriter.
Also to protect us in the future and make sure close method is never called from inside Translog special assertion examining stack trace is added. Since we're still targeting Java 8 for runtime - no StackWalker API is used in the implementation.
In the stack-trace checking method, we're considering inner caller not only Translog methods but Translog child classes methods as well. It does mean that Translog is meant for extending it, but it's needed to be able to test this method.
Closes#32526
This is related to #32517. This commit passes the DiscoveryNode to the
initiateChannel method for different Transport implementation. This
will allow additional attributes (besides just the socket address) to be
used when opening channels.
Randomized test conditions that cause some shards to have no docs on them
failed due to test asserts that relied on a lazy initialization side effect
from the map script. After this fix:
- Test cases with the relevant init script are protected
- Test cases with the relevant combine or reduce scripts were already
protected, because the combine and reduce scripts safely handle this case.
This is a followup to #31886. After that commit the
TransportConnectionListener had to be propogated to both the
Transport and the ConnectionManager. This commit moves that listener
to completely live in the ConnectionManager. The request and response
related methods are moved to a TransportMessageListener. That listener
continues to live in the Transport class.
* elastic/master: (46 commits)
NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS (#32764)
[DOCS] Splits the users API documentation into multiple pages (#32825)
[DOCS] Splits the token APIs into separate pages (#32865)
[DOCS] Creates redirects for role management APIs page
Bypassing failing test PainlessDomainSplitIT#testHRDSplit (#32966)
TEST: Mute testRetentionPolicyChangeDuringRecovery
[DOCS] Fixes more broken links to role management APIs
[Docs] Tweaks and fixes to rollup docs
[DOCS] Fixes links to role management APIs
[ML][TEST] Fix BasicRenormalizationIT after adding multibucket feature
[DOCS] Splits the roles API documentation into multiple pages (#32794)
[TEST] Run pre 6.4 nodes in non-FIPS JVMs (#32901)
Make Geo Context Mapping Parsing More Strict (#32821)
[ML] fix updating opened jobs scheduled events (#31651) (#32881)
Scripted metric aggregations: add deprecation warning and system property to control legacy params (#31597)
Tests: Fix timezone conversion in DateTimeUnitTests
Enable FIPS140LicenseBootstrapCheck (#32903)
Fix InternalAutoDateHistogram reproducible failure (#32723)
Remove assertion in testDocStats on deletedDocs counter (#32914)
HLRC: Move ML request converters into their own class (#32906)
...
* Lazy resolve DNS (i.e. `String` to `DiscoveryNode`) to not run into indefinitely caching lookup issues (provided the JVM dns cache is configured correctly as explained in https://www.elastic.co/guide/en/elasticsearch/reference/6.3/networkaddress-cache-ttl.html)
* Changed `InetAddress` type to `String` for that higher up the stack
* Passed down `Supplier<DiscoveryNode>` instead of outright `DiscoveryNode` from `RemoteClusterAware#buildRemoteClustersSeeds` on to lazy resolve DNS when the `DiscoveryNode` is actually used (could've also passed down the value of `clusterName = REMOTE_CLUSTERS_SEEDS.getNamespace(concreteSetting)` together with the `List<String>` of hosts, but this route seemed to introduce less duplication and resulted in a significantly smaller changeset).
* Closes#28858
Currently, if geo context is represented by something other than
geo_point or an object with lat and lon fields, the parsing of it
as a geo context can result in ignoring the context altogether,
returning confusing errors such as number_format_exception or trying
to parse the number specifying as long-encoded hash code. It would also
fail if the geo_point was stored.
This commit makes the mapping parsing more strict and will fail during
mapping update or index creation if the geo context doesn't point to
a geo_point field.
Supersedes #32412Closes#32202
* Scripted metric aggregations: add deprecation warning and system property to control legacy params
Scripted metric aggregation params._agg/_aggs are replaced by state/states context variables. By default the old params are still present, and a deprecation warning is emitted when Scripted Metric Aggregations are used. A new system property can be used to disable the legacy params. This functionality will be removed in a future revision.
* Fix minor style issue and docs test failure
* Disable deprecated params._agg/_aggs in tests and revise tests to use state/states instead
* Add integration test covering deprecated scripted metrics aggs params._agg/_aggs access
* Disable deprecated params._agg/_aggs in docs integration tests and revise stored scripts to use state/states instead
* Revert unnecessary migrations doc change
A relevant note should be added in the changes destined for 7.0; this PR is going to be backported to 6.x.
* Replace deprecated _agg param bwc integration test with a couple of unit tests
* Fix compatibility test after merge
* Rename backwards compatibility system property per code review feedback
* Tweak deprecation warning text per review feedback
This fix prevernts trying to parse unknown timezone ids by converting
the joda time zone via java.util.TimeZone to a java time based ZoneId.
Closes#32927
testDocStats test is flaky and sometimes it's failing on jenkins and
failure is not reproducible locally. The reason for this failure is in
timing. If the number of deleted documents is greater than 33% of inserted
documents, Lucene will schedule segments to merge if TieredMergePolicy is
used (it's not the case for LogMergePolicy, but ES is only using
TieredMergePolicy). If this merge is performed before stats are
retrieved - we will get 0 for "deleted" counter.
So basically this counter could be either 0 or numOfDeletedDocs at this point,
but this is the too loose assertion and we decided to remove it at all.
Closes#32766
This commit disables the automatic `refresh_interval` in order to ensure
that index readers cannot differ between the normal and scroll search.
This issue is related to the 7.5 Lucene upgrade which contains a change that
makes single segment merge more likely to occur (max deletes percentage).
Closes#32682
We do not support passphrases on the secure settings storage (the
keystore). Yet, we added support for this in the API layer. This commit
removes this support so that we are not limited in our future options,
or have to make a breaking change.
This change cleans up some methods in the CharArrays class from x-pack, which
includes the unification of char[] to utf8 and utf8 to char[] conversions that
intentionally do not use strings. There was previously an implementation in
x-pack and in the reloading of secure settings. The method from the reloading
of secure settings was adopted as it handled more scenarios related to the
backing byte and char buffers that were used to perform the conversions. The
cleaned up class is moved into libs/core to allow it to be used by requests
that will be migrated to the high level rest client.
Relates #32332
* master:
Fix global checkpoint listeners test
HLRC: adding machine learning open job (#32860)
[ML] Add log structure finder functionality (#32788)
INGEST: Add Configuration Except. Data to Metdata (#32322)
This commit fixes a global checkpoint listeners test wherein we were
expecting an executor to have been used even if there were no
listeners. This is silliness, so this commit adjusts the assertion to
verify that the executor never fires if there are no listeners, and
fires exactly once if there is one or more listeners.
This commit introduces the ability for global checkpoint listeners to be
registered at the shard level. These listeners are notified when the
global checkpoint is updated, and also when the shard closes. To
encapsulate these listeners, we introduce a shard-level component that
handles synchronization of notification and modifications to the
collection of listeners.
This is related to #31835. It moves the default connection profile into
the ConnectionManager class. The will allow us to have different
connection managers with different profiles.
This removes custom Response classes that extend `AcknowledgedResponse` and do nothing, these classes are not needed and we can directly use the non-abstract super-class instead.
While this appears to be a large PR, no code has actually changed, only class names have been changed and entire classes removed.
This commit adds a java time version of the existing rounding classes, which features the same test suite and a small test class to check if serialization works as expected.
Significance score doubles were being parsed as long. Existing tests did not catch this because SignificantLongTermsTests and SignificantStringTermsTests did not set the score. Fixed these and also added integration test.
Thanks for the report/fix, Blakko
Closes#32770
* INGEST: Create Index Before Pipeline Execute
* Ensures that indices are created before the default pipeline setting is read to correcly handle the case of an index template containing a default pipeline (without the fix the first document does not get the pipeline applied as explained in #32758)
* closes#32758
#31821 introduced an unreleased bug where NOOP updates were incorrectly mutating the bulk
shard request, inserting null item to be replicated, which would result in NullPointerExceptions when
serializing the request to be shipped to the replicas.
Closes#32808
This is related to #31835. This commit adds a connection manager that
manages client connections to other nodes. This means that the
TcpTransport no longer maintains a map of nodes that it is connected
to.
With the move to java time, the default formatter used by toString on
ZonedDateTime uses optional components for least significant portions of
the date. This commit changes the cat indices api to use a strict date
time format, which will always output milliseconds, even if they are
zero.
closes#32466
Currently AbstractBuilderTestCase generates certain random values in its
`beforeTest()` method annotated with @Before only the first time that a test
method in the suite is run while initializing the serviceHolder that we use for
the rest of the test. This changes the values of subsequent random values
and has the effect that when running single methods from a test suite with
"-Dtests.method=*", the random values it sees are different from when the same
test method is run as part of the whole test suite. This makes it hard to use
the reproduction lines logged on failure.
This change runs the inialization of the serviceHolder and the randomization
connected to it using the test runners master seed, so reproduction by running
just one method is possible again.
Closes#32400
Processing bulk request goes item by item. Sometimes during processing, we need to stop execution and wait for a new mapping update to be processed by the node. This is currently achieved by throwing a `RetryOnPrimaryException`, which is caught higher up. When the exception is caught, we wait for the next cluster state to arrive and process the request again. Sadly this is a problem because all operations that were already done until the mapping change was required are applied again and get new sequence numbers. This in turn means that the previously issued sequence numbers are never replicated to the replicas. That causes the local checkpoint of those shards to be stuck and with it all the seq# based infrastructure.
This commit refactors how we deal with retries with the goal of removing `RetryOnPrimaryException` and `RetryOnReplicaException` (not done yet). It achieves so by introducing a class `BulkPrimaryExecutionContext` that is used the capture the execution state and allows continuing from where the execution stopped. The class also formalizes the steps each item has to go through:
1) A translation phase for updates
2) Execution phase (always index/delete)
3) Waiting for a mapping update to come in, if needed
4) Requires a retry (for updates and cases where the mapping are still not available after the put mapping call returns)
5) A finalization phase which allows updates to the index/delete result to an update result.
This adds a java time based date math parser class in order, which will replace the joda date based one in the future. For now the class also returns the date in milliseconds since the epoch.
Currently if a document cannot be indexed because it violates the defined
mapping for the index, a MapperException is thrown. In some cases it is
useful to expose the expected field type in the exception itself,
so that the user can react based on the error message. This change adds
the expected data type to the MapperException.
Closes#31502
Remove a few of the logger constructors that aren't widely used or
aren't used at all and deprecate a few more logger constructors in favor
of log4j2's `LogManager`.
A bug in the test suite prevented to properly check that all date
formatters printed the date the same way like joda time does.
This fixes the test and thus also a fair share of formats, that
now use the strict parser for printing.
We previously discussed moving the classes extending `AcknowledgedResponse` to
simply use `AcknowledgedResponse`, making the class non-abstract.
This moves the first class to do this, removing `WritePipelineResponse` in the
process.
If we like the way this looks, I will switch the remaining classes over to using
`AcknowledgedResponse`.
When Circuit Breaker has tripped, certain diagnostic requests like
"_cluster/health" succeed where as request to / fails with
503 Service Unavailable. This behavior is observed because of this
commit f32b700 where certain API paths are whitelisted from
Circuit Breaking exception, but / is not whitelisted.
Added / to circuit breaker whitelist so that it can be used for
diagnostic purposes
* Fixes suggestion generics
This solves a compile problem in Eclipse where Eclipse could not
resolve the generics for the options field in `PhraseSuggestion.Entry`.
But I think this is also a good change in general because
`PhraseSuggestion.Entry` is now declaring the specific `Option`
implementation it requires rather than `Suggest.Entry.Option` which is
more general and could lead to weird bugs. `CompletionSuggestion.Entry`
and `TermSuggestion.Entry` already declare the more specific class they
use so I think this was an oversight in `PhaseSuggestion.Entry`
* iter
`ShardOperationFailedException` and corresponding implementors seem to suggest that the cause may be null, case that is also handled in a few places. Yet, it does not seem to be possible in practice for the cause to be null, hence we can clean that up and enforce the cause to be a non null value. This is best done by making `ShardOperationFailedException` an abstract class rather than an interface, which holds the basic member instance that all the subclasses have in common and can also enforce that cause, status and reason are non null.
As part of #32608 we made sure that the fully qualified index name is taken from the query shard context whenever creating a new `QueryShardException`. That change introduced a regression as instead of setting the entire `Index` object to the exception, which holds index name and index uuid, we ended up setting only the index name (including cluster alias). With this commit we make sure that the index uuid does not get lost and we try to lower the chances that a similar bug makes it in another time. That's done by making `QueryShardContext` return the fully qualified `Index` (which also holds the uuid) rather than only the fully qualified index name.
Suggestion responses were previously serialized as streamables which
made writing suggesters in plugins with custom suggestion response types
impossible. This commit makes them serialized as named writeables and
provides a facility for registering a reader for suggestion responses
when registering a suggester.
This also makes Suggestion responses abstract, requiring a suggester
implementation to provide its own types. Suggesters which do not need
anything additional to what is defined in Suggest.Suggestion should
provide a minimal subclass.
The existing plugin suggester integration tests are removed and
replaced with an equivalent implementation as an example
plugin.
It will be useful for future efforts to know if the global checkpoint
was updated. To this end, we need to expose whether or not the global
checkpoint was updated when the state of the replication tracker
updates. For this, we add to the tracker a callback that is invoked
whenever the global checkpoint is updated. For primaries this will be
invoked when the computed global checkpoint is updated based on state
changes to the tracker. For replicas this will be invoked when the local
knowledge of the global checkpoint is advanced from the primary.
The MockNioTransport (similar to the MockTcpTransport) is used for integ
tests. The MockTcpTransport has always only opened a single for all of
its work. The MockNioTransport has awlays opened the default number of
connections (13). This means that every test where two transports
connect requires 26 connections. This is more than is necessary. This
commit modifies the MockNioTransport to only require 3 connections.
Primary terms were introduced as part of the sequence-number effort (#10708) and added in ES
5.0. Subsequent work introduced the replication tracker which lets the primary own its replication
group (#25692) to coordinate recovery and replication. The replication tracker explicitly exposes
whether it is operating in primary mode or replica mode, independent of the ShardRouting object
that's associated with a shard. During a primary relocation, for example, the primary mode is
transferred between the primary relocation source and the primary relocation target. After
transferring this so-called primary context, the old primary becomes a replication target and the
new primary the replication source, reflected in the replication tracker on both nodes. With the
most recent PR in this area (#32442), we finally have a clean transition between a shard that's
operating as a primary and issuing sequence numbers and a shard that's serving as a replication
target. The transition from one state to the other is enforced through the operation-permit system,
where we block permit acquisition during such changes and perform the transition under this
operation block, ensuring that there are no operations in progress while the transition is being
performed. This finally allows us to turn the best-effort checks that were put in place to prevent
shards from being used in the wrong way (i.e. primary as replica, or replica as primary) into hard
assertions, making it easier to catch any bugs in this area.
Currently, when TranslogCorruptedException is thrown most of the times it does not contain information about the translog location on the file system. There is the translog recovery tool that accepts the translog path as an argument and users are constantly puzzled where to get the path.
This pull request adds "source" information to every TranslogCorruptedException thrown. The source could be local file, remote translog source (used for recovery), assertion (translog entry is constructed to perform some assertion) or translog constructed inside the test.
Closes#24929
This change adds a check so that when parsing the search source, script fields are
ignored when the requested search result size is 0. This helps with e.g. clients like
Kibana that sends a list of script fields that they may need for convenience, but they
don't require any hits. Before this change, user sometimes ran into confusing behaviour,
e.g. the script compilation limit to breaking although no hits were requested.
Closes#31824
* We were comparing the wrong timeout value in the `randomValueOtherThan` call here, leading to no mutation happening for a certain seed
* closes#32639
Today content type detection on an input stream works by peeking up to
twenty bytes into the stream. If the stream is headed by more whitespace
than twenty bytes, we might fail to detect the content type. We should
be ignoring this whitespace before attempting to detect the content
type. This commit does that by ignoring all leading whitespace in an
input stream before attempting to guess the content type.
The assertion in the test was not broad enough. If the timing is very unlucky, the
shard is already promoted to primary before the indexOnReplica even gets to execute.
Closes#32645
* master:
Cross-cluster search: preserve cluster alias in shard failures (#32608)
Handle AlreadyClosedException when bumping primary term
[TEST] Allow to run in FIPS JVM (#32607)
[Test] Add ckb to the list of unsupported languages (#32611)
SCRIPTING: Move Aggregation Scripts to their own context (#32068)
Painless: Use LocalMethod Map For Lookup at Runtime (#32599)
[TEST] Enhance failure message when bulk updates have failures
[ML] Add ML result classes to protocol library (#32587)
Suppress LicensingDocumentationIT.testPutLicense in release builds (#32613)
[Rollup] Update wire version check after backport
Suppress Wildfly test in FIPS JVMs (#32543)
[Rollup] Improve ID scheme for rollup documents (#32558)
ingest: doc: move Dot Expander Processor doc to correct position (#31743)
[ML] Add some ML config classes to protocol library (#32502)
[TEST]Split transport verification mode none tests (#32488)
Core: Move helper date formatters over to java time (#32504)
[Rollup] Remove builders from DateHistogramGroupConfig (#32555)
[TEST} unmutes SearchAsyncActionTests and adds debugging info
[ML] Add Detector config classes to protocol library (#32495)
[Rollup] Remove builders from MetricConfig (#32536)
Tests: Add rolling upgrade tests for watcher (#32428)
Fix race between replica reset and primary promotion (#32442)
When some remote clusters return shard failures as part of a cross-cluster search request, the cluster alias currently gets lost. As a result, if the shard failures are all caused by the same error, and against indices belonging to different clusters, but with the same index name, only one failure gets returned as part of the search response, meaning that failures are grouped by index name, ignoring the cluster alias.
With this commit we make sure that `ShardSearchFailure` returns the cluster alias as part of the index name. Also, we set the fully qualfied index name when creating a `QueryShardException`. That way shard failures are grouped by cluster:index. Such fixes should cover at least most of the cases where either 1) the shard target is set but we don't have the index in the cause (we were previously reading it only from the cause that did not have the cluster alias) 2) the shard target is missing but if the cause is a `QueryShardException` the cluster alias does not get lost.
We also prevent NPE in case the failure cause is not set and test such scenario.
If the shard is already closed while bumping the primary term, this can result in an
AlreadyClosedException to be thrown. As we use asyncBlockOperations, the exception
will be thrown on a thread from the generic thread pool and end up in the uncaught
exception handler, failing our tests.
Relates to #32442
Some classes use internal date formatters, which now can be moved over
to java time using the DateFormatters class.
The same applies for a few test cases.
This unmutes the testFanOutAndCollect()` method and add a check to make
sure we aren't accidentally running something twice causing a search
phase to still be running after we have counted down the latch
Relates to #29242
We've recently seen a number of test failures that tripped an assertion in IndexShard (see issues
linked below), leading to the discovery of a race between resetting a replica when it learns about a
higher term and when the same replica is promoted to primary. This commit fixes the race by
distinguishing between a cluster state primary term (called pendingPrimaryTerm) and a shard-level
operation term. The former is set during the cluster state update or when a replica learns about a
new primary. The latter is only incremented under the operation block, which can happen in a
delayed fashion. It also solves the issue where a replica that's still adjusting to the new term
receives a cluster state update that promotes it to primary, which can happen in the situation of
multiple nodes being shut down in short succession. In that case, the cluster state update thread
would call `asyncBlockOperations` in `updateShardState`, which in turn would throw an exception
as blocking permits is not allowed while an ongoing block is in place, subsequently failing the shard.
This commit therefore extends the IndexShardOperationPermits to allow it to queue multiple blocks
(which will all take precedence over operations acquiring permits). Finally, it also moves the primary
activation of the replication tracker under the operation block, so that the actual transition to
primary only happens under the operation block.
Relates to #32431, #32304 and #32118
* master:
HLRC: Move commercial clients from XPackClient (#32596)
Add cluster UUID to Cluster Stats API response (#32206)
Security: move User to protocol project (#32367)
[TEST] Test for shard failures, add debug to testProfileMatchesRegular
Minor fix for javadoc (applicable for java 11). (#32573)
Painless: Move Some Lookup Logic to PainlessLookup (#32565)
TEST: Avoid merges in testSeqNoAndCheckpoints
[Rollup] Remove builders from HistoGroupConfig (#32533)
Mutes failing SQL string function tests due to #32589
fixed elements in array of produced terms (#32519)
INGEST: Enable default pipelines (#32286)
Remove cluster state initial customs (#32501)
Mutes LicensingDocumentationIT due to #32580
[ML] Remove multiple_bucket_spans (#32496)
[ML] Rename JobProvider to JobResultsProvider (#32551)
Correct minor typo in explain.asciidoc for HLRC
Build: Add elastic maven to repos used by BuildPlugin (#32549)
Clarify the error message when a pipeline agg is used in the 'order' parameter. (#32522)
Revert "[test] turn on host io cache for opensuse (#32053)"
Enable packaging tests on suse boxes
[ML] Improve error when no available field exists for rule scope (#32550)
[ML] Improve error for functions with limited rule condition support (#32548)
Painless: Clean Up PainlessField (#32525)
Add @AwaitsFix for #32554
Remove broken @link in Javadoc
Scripting: Conditionally use java time api in scripting (#31441)
[ML] Fix thread leak when waiting for job flush (#32196) (#32541)
Add AwaitsFix to failing test - see #32546
Core: Minor size reduction for AbstractComponent (#32509)
SQL: Added support for string manipulating functions with more than one parameter (#32356)
[DOCS] Reloadable Secure Settings (#31713)
Watcher: Reenable HttpSecretsIntegrationTests#testWebhookAction test (#32456)
[Rollup] Remove builders from TermsGroupConfig (#32507)
Use hostname instead of IP with SPNEGO test (#32514)
Switch x-pack rolling restart to new style Requests (#32339)
NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
[DOCS] Small fixes in rule configuration page (#32516)
Painless: Clean up PainlessMethod (#32476)
Build: Remove shadowing from benchmarks (#32475)
Docs: Add all JDKs to CONTRIBUTING.md
Add licensing enforcement for FIPS mode (#32437)
SQL: Add test for handling of partial results (#32474)
Mute testFilterCacheStats
[ML][DOCS] Fix typo applied_to => applies_to
Scripting: Fix painless compiler loader to know about context classes (#32385)
* Make cluster stats response contain cluster UUID
* Updating constructor usage in Monitoring tests
* Adding cluster_uuid field to Cluster Stats API reference doc
* Adding rest api spec test for expecting cluster_uuid in cluster stats response
* Adding missing newline
* Indenting do section properly
* Missed a spot!
* Fixing the test cluster ID
Unmuting the test and adding some more debug output. Was not able to
reproduce the prior failure, but it seems possible that the
failure (mismatched counts) could be caused by partial search results
during the test.
The assertions check for shard failures first, because if one of the
two searches is partial the rest of the test will fail.
Next, instead of just checking respective hit counts, we emit the
difference in hits to help identify what went wrong.
Closes#32492
Since LUCENE-8263, testRecoveryWithOutOfOrderDelete may trigger merges
because of the deletes. In the test, we try to retain index#0 but
reclaim delete#1. However, if a merge is triggered, we will remove both
index#0 and delete#1.
This commit disables merges in this test. Another option is to index
more documents in the segment_2 to reduce the deletion ratio.
Since LUCENE-8263, testSeqNoAndCheckpoints might trigger merges because
of the updates and deletes in the test. Our merge scheduler will trigger
a flush if there is no pending merge. Those extra flushes will change
the last committed segmentInfos in the engine and fail the test.
This commit uses LogMergePolicy for the engine in the test to avoid
merges.
Closes#32430
This commit adds a boolean system property, `es.scripting.use_java_time`,
which controls the concrete return type used by doc values within
scripts. The return type of accessing doc values for a date field is
changed to Object, essentially duck typing the type to allow
co-existence during the transition from joda time to java time.
This removes a constructor from `AbstractComponent` and
`AbstractLifecycleComponent` that we weren't using and it switches the
logger creation away from one of the `Settings` flavored methods which
are no longer needed.
First, some background: we have 15 different methods to get a logger in
Elasticsearch but they can be broken down into three broad categories
based on what information is provided when building the logger.
Just a class like:
```
private static final Logger logger = ESLoggerFactory.getLogger(ActionModule.class);
```
or:
```
protected final Logger logger = Loggers.getLogger(getClass());
```
The class and settings:
```
this.logger = Loggers.getLogger(getClass(), settings);
```
Or more information like:
```
Loggers.getLogger("index.store.deletes", settings, shardId)
```
The goal of the "class and settings" variant is to attach the node name
to the logger. Because we don't always have the settings available, we
often use the "just a class" variant and get loggers without node names
attached. There isn't any real consistency here. Some loggers get the
node name because it is convenient and some do not.
This change makes the node name available to all loggers all the time.
Almost. There are some caveats are testing that I'll get to. But in
*production* code the node name is node available to all loggers. This
means we can stop using the "class and settings" variants to fetch
loggers which was the real goal here, but a pleasant side effect is that
the ndoe name is now consitent on every log line and optional by editing
the logging pattern. This is all powered by setting the node name
statically on a logging formatter very early in initialization.
Now to tests: tests can't set the node name statically because
subclasses of `ESIntegTestCase` run many nodes in the same jvm, even in
the same class loader. Also, lots of tests don't run with a real node so
they don't *have* a node name at all. To support multiple nodes in the
same JVM tests suss out the node name from the thread name which works
surprisingly well and easy to test in a nice way. For those threads
that are not part of an `ESIntegTestCase` node we stick whatever useful
information we can get form the thread name in the place of the node
name. This allows us to keep the logger format consistent.
When using cross-cluster search through the high-level REST client, the cluster alias from each search hit was not parsed correctly. It would be part of the index field initially, but overridden just a few lines later once setting the shard target (in case we have enough info to build it from the response). In any case, getClusterAlias returns `null` which is a bug.
With this change we rather parse back clusterAliases from the index name, set its corresponding field and properly handle the two possible cases depending on whether we can or cannot build the shard target object.
The method for working out whether a polygon is clockwise or anticlockwise is
mostly correct but doesn't work in some rare cases such as the included test
case. This commit fixes that.
Rollover should not swap aliases when `is_write_index` is set to `true`.
Instead, both the new and old indices should have the rollover alias,
with the newly created index as the new write index
Updates Rollover to leverage the ability to preserve aliases and swap which is the write index.
Historically, Rollover would swap which index had the designated alias for writing documents against. This required users to keep a separate read-alias that enabled reading against both rolled over and newly created indices, whiles the write-alias was being re-assigned at every rollover.
With the ability for aliases to designate a write index, Rollover can be a bit more flexible with its use of aliases.
Updates include:
- Rollover validates that the target alias has a write index (the index that is being rolled over). This means that the restriction that aliases only point to one index is no longer necessary.
- Rollover explicitly (and atomically) swaps which index is the write-index by explicitly assigning the existing index to have `is_write_index: false` and have the newly created index have its rollover alias as `is_write_index: true`. This is only done when `is_write_index: true` on the write index. Default behavior of removing the alias from the rolled over index stays when `is_write_index` is not explicitly set
Relevant things that are staying the same:
- Rollover is rejected if there exist any templates that match the newly-created index and configure the rollover-alias
- I think this existed to prevent the situation where an alias pointed to two indices for a short while. Although this can technically be relaxed, the specific cases that are safe are really particular and difficult to reason, so leaving the broad restriction sounds good
* Ensure decryption related exceptions are handled
This commit ensures that all possible Exceptions in
KeyStoreWrapper#decrypt() are handled. More specifically, in the
case that a wrong password is used for secure settings, calling readX
on the DataInputStream that wraps the CipherInputStream can throw an
IOException. It also adds a test for loading a KeyStoreWrapper with
a wrong password.
Resolves#32411
Today when reading operation history in Lucene, we read all documents.
However, if indexing a document is aborted, IndexWriter will hard-delete
it; we, therefore, need to exclude that document from Lucene history.
This commit makes sure that we exclude aborted documents by using the
hard liveDocs of a SegmentReader if there are deletes.
Closes#32269
* master:
Tests: Fix convert error tests to use fixed value (#32415)
IndicesClusterStateService should replace an init. replica with an init. primary with the same aId (#32374)
REST high-level client: parse back _ignored meta field (#32362)
[CI] Mute DocumentSubsetReaderTests testSearch
In rare cases it is possible that a nodes gets an instruction to replace a replica
shard that's in `POST_RECOVERY` with a new initializing primary with the same allocation id.
This can happen by batching cluster states that include the starting of the replica, with
closing of the indices, opening it up again and allocating the primary shard to the node in
question. The node should then clean it's initializing replica and replace it with a new
initializing primary.
I'm not sure whether the test I added really adds enough value as existing tests found this. The main reason I added is to allow for simpler reproduction and to double check I fixed it. I'm open to discuss if we should keep.
Closes#32308
`GetResult` and `SearchHit` have been adjusted to parse back the `_ignored` meta field whenever it gets printed out. Expanded the existing tests to make sure this is covered. Fixed also a small problem around highlighted fields in `SearchHitTests`.
* master:
TEST: testDocStats should always use forceMerge (#32450)
TEST: Avoid deletion in FlushIT
AwaitsFix IndexShardTests#testDocStats
Painless: Add method type to method. (#32441)
Due to the recent change in LUCENE-8263, we need to adjust the deletion
ration to between 10% to 33% to preserve the current behavior of the
test. However, we may need another refinement if soft-deletes is enabled
as the actual deletes are different because of delete tombstones.
This commit prefers to always execute forceMerge instead of adjusting
the deletion ratio so that this test can focus on testing docStats.
Closes#32449
Due to the recent change in LUCENE-8263, a merge can be triggered if the
deletion ration is higher than 33%. An in-progress merge can prevent a
synced-flush from issuing.
This commit avoids deletes by using different docIds.
Closes#32436
* master:
Remove reference to non-existent store type (#32418)
[TEST] Mute failing FlushIT test
Fix ordering of bootstrap checks in docs (#32417)
[TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints
[TEST] Mute failing testConvertLongHexError
bump lucene version after backport
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
[Kerberos] Avoid vagrant update on precommit (#32416)
TESTS: Move netty leak detection to paranoid level (#32354)
[DOCS] Fixes formatting of scope object in job resource
Copy missing segment attributes in getSegmentInfo (#32396)
AbstractQueryTestCase should run without type less often (#28936)
INGEST: Fix Deprecation Warning in Script Proc. (#32407)
Switch x-pack/plugin to new style Requests (#32327)
Docs: Correcting a typo in tophits (#32359)
Build: Stop double generating buildSrc pom (#32408)
TEST: Avoid triggering merges in FlushIT
Fix missing JavaDoc for @throws in several places in KerberosTicketValidator.
Switch x-pack full restart to new style Requests (#32294)
Release requests in cors handler (#32364)
Painless: Clean Up PainlessClass Variables (#32380)
Docs: Fix callouts in put license HL REST docs (#32363)
[ML] Consistent pattern for strict/lenient parser names (#32399)
Update update-settings.asciidoc (#31378)
Remove some dead code (#31993)
Introduce index store plugins (#32375)
Rank-Eval: Reduce scope of an unchecked supression
Make sure _forcemerge respects `max_num_segments`. (#32291)
TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests (#32377)
Only enforce password hashing check if FIPS enabled (#32383)
Today it's possible to encounter an Index operation in Lucene whose
_source is disabled, and _recovery_source was pruned by the MergePolicy.
If it's the case, we create a Translog#Index without source and let the
caller validate it later. However, this approach is challenging for the
caller.
Deletes and No-Ops don't allow invoking "source()" method. The caller
has to make sure to call "source()" only on index operations. The
current implementation in CCR does not follow this and fail to replica
deletes or no-ops. Moreover, it's easier to reason if a Translog#Index
always has the source.
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
This commit changes the randomization to always create an index with a type.
It also adds a way to create a query shard context that maps to an index with
no type registered in order to explicitely test cases where there is no type.
* Using short script form normalized to a map that used 'inline' instead of 'source' so a short form processor definition like:
```
{
"script": "ctx.foo= 'bar'"
}
```
would always warn about the following deprecation:
```
#! Deprecation: Deprecated field [inline] used, expected [source]
```
In testSyncedFlushSkipOutOfSyncReplicas, we reindex the extra documents
to all shards including the out-of-sync replica. However, reindexing to
that replica can trigger merges (due to the new deletes) which cause the
synced-flush failed. This test starts failing after we aggressively
trigger merges segments with a large number of deletes in LUCENE-8263.
Removing some dead code or supressing warnings where apropriate. Most of the
time the variable tested for null is dereferenced earlier or never used before.
Today we allow plugins to add index store implementations yet we are not
doing this in our new way of managing plugins as pull versus push. That
is, today we still allow plugins to push index store providers via an on
index module call where they can turn around and add an index
store. Aside from being inconsistent with how we manage plugins today
where we would look to pull such implementations from plugins at node
creation time, it also means that we do not know at a top-level (for
example, in the indices service) which index stores are available. This
commit addresses this by adding a dedicated plugin type for index store
plugins, removing the index module hook for adding index stores, and by
aggregating these into the top-level of the indices service.
An upcoming [Lucene change](https://issues.apache.org/jira/browse/LUCENE-7976)
will make TieredMergePolicy respect the maximum merged segment size all the
time, meaning it will possibly not respect the `max_num_segments` parameter
anymore if the shard is larger than the maximum segment size.
This change makes sure that `max_num_segments` is respected for now in order
to give us time to think about how to integrate this change, and also to delay
it until 7.0 as this might be a big-enough change for us to wait for a new
major version.
* master:
[DOCS] Fix formatting error in Slack action
Painless: Fix documentation links to use existing refs (#32335)
Painless: Decouple PainlessLookupBuilder and Whitelists (#32346)
[DOCS] Adds recommendation for xpack.security.enabled (#32345)
[TEST] Mute ConvertProcessortTests.testConvertIntHexError
[TEST] Fix failure due to exception message in java11 (#32321)
[DOCS] Fixes typo in ML aggregations page
[DOCS] Adds link from bucket_span property to common time units
[ML][DOCS] Add documentation for detector rules and filters (#32013)
Add opaque_id to index audit logging (#32260)
Add 6.5.0 version to master
fixes broken build for third-party-tests (#32353)
* master:
Security: revert to old way of merging automata (#32254)
Networking: Fix test leaking buffer (#32296)
Undo a debugging change that snuck in during the field aliases merge.
Painless: Update More Methods to New Naming Scheme (#32305)
[TEST] Fix assumeFalse -> assumeTrue in SSLReloadIntegTests
Ingest: Support integer and long hex values in convert (#32213)
Introduce fips_mode setting and associated checks (#32326)
Add V_6_3_3 version constant
[DOCS] Removed extraneous callout number.
Rest HL client: Add put license action (#32214)
Add ERR to ranking evaluation documentation (#32314)
Introduce Application Privileges with support for Kibana RBAC (#32309)
Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
[Kerberos] Add Kerberos authentication support (#32263)
[ML] Extract persistent task methods from MlMetadata (#32319)
Add Restore Snapshot High Level REST API
Register ERR metric with NamedXContentRegistry (#32320)
fixes broken build for third-party-tests (#32315)
Allow Integ Tests to run in a FIPS-140 JVM (#31989)
[DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280)
awaitsfix testRandomClusterStateUpdates
[TEST] add version skip to weighted_avg tests
Consistent encoder names (#29492)
Add WeightedAvg metric aggregation (#31037)
Switch monitoring to new style Requests (#32255)
Rename ranking evaluation `quality_level` to `metric_score` (#32168)
Fix a test bug around nested aggregations and field aliases. (#32287)
Add new permission for JDK11 to load JAAS libraries (#32132)
Silence SSL reload test that fails on JDK 11
[test] package pre-install java check (#32259)
specify subdirs of lib, bin, modules in package (#32253)
Switch x-pack:core to new style Requests (#32252)
awaitsfix SSLConfigurationReloaderTests
Painless: Clean up add methods in PainlessLookup (#32258)
Fail shard if IndexShard#storeStats runs into an IOException (#32241)
AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated
Remove unnecessary warning supressions (#32250)
CCE when re-throwing "shard not available" exception in TransportShardMultiGetAction (#32185)
Add new fields to monitoring template for Beats state (#32085)
* Introduce fips_mode setting and associated checks
Introduce xpack.security.fips_mode.enabled setting ( default false)
When it is set to true, a number of Bootstrap checks are performed:
- Check that Secure Settings are of the latest version (3)
- Check that no JKS keystores are configured
- Check that compliant algorithms ( PBKDF2 family ) are used for
password hashing
This commit introduces "Application Privileges" to the X-Pack security
model.
Application Privileges are managed within Elasticsearch, and can be
tested with the _has_privileges API, but do not grant access to any
actions or resources within Elasticsearch. Their purpose is to allow
applications outside of Elasticsearch to represent and store their own
privileges model within Elasticsearch roles.
Access to manage application privileges is handled in a new way that
grants permission to specific application names only. This lays the
foundation for more OLS on cluster privileges, which is implemented by
allowing a cluster permission to inspect not just the action being
executed, but also the request to which the action is applied.
To support this, a "conditional cluster privilege" is introduced, which
is like the existing cluster privilege, except that it has a Predicate
over the request as well as over the action name.
Specifically, this adds
- GET/PUT/DELETE actions for defining application level privileges
- application privileges in role definitions
- application privileges in the has_privileges API
- changes to the cluster permission class to support checking of request
objects
- a new "global" element on role definition to provide cluster object
level security (only for manage application privileges)
- changes to `kibana_user`, `kibana_dashboard_only_user` and
`kibana_system` roles to use and manage application privileges
Closes#29820Closes#31559
* Complete changes for running IT in a fips JVM
- Mute :x-pack:qa:sql:security:ssl:integTest as it
cannot run in FIPS 140 JVM until the SQL CLI supports key/cert.
- Set default JVM keystore/truststore password in top level build
script for all integTest tasks in a FIPS 140 JVM
- Changed top level x-pack build script to use keys and certificates
for trust/key material when spinning up clusters for IT
Adds a new single-value metrics aggregation that computes the weighted
average of numeric values that are extracted from the aggregated
documents. These values can be extracted from specific numeric
fields in the documents.
When calculating a regular average, each datapoint has an equal "weight"; it
contributes equally to the final value. In contrast, weighted averages
scale each datapoint differently. The amount that each datapoint contributes
to the final value is extracted from the document, or provided by a script.
As a formula, a weighted average is the `∑(value * weight) / ∑(weight)`
A regular average can be thought of as a weighted average where every value has
an implicit weight of `1`.
Closes#15731
ClassCastException can be thrown by callers of TransportActions.isShardNotAvailableException(e) as e is not always an instance of ElasticSearchException
fixes#32173
* es/master: (23 commits)
Switch full-cluster-restart to new style Requests (#32140)
[DOCS] Clarified that you must remove X-Pack plugin when upgrading from pre-6.3. (#32016)
Remove BouncyCastle dependency from runtime (#32193)
INGEST: Extend KV Processor (#31789) (#32232)
INGEST: Make a few Processors callable by Painless (#32170)
Add region ISO code to GeoIP Ingest plugin (#31669)
[Tests] Remove QueryStringQueryBuilderTests#toQuery class assertions (#32236)
Make sure that field aliases count towards the total fields limit. (#32222)
Switch rolling restart to new style Requests (#32147)
muting failing test for internal auto date histogram to avoid failure before fix is merged
MINOR: Remove unused `IndexDynamicSettings` (#32237)
Fix multi level nested sort (#32204)
Enhance Parent circuit breaker error message (#32056)
[ML] Use default request durability for .ml-state index (#32233)
Remove indices stats timeout from monitoring docs
Rename ranking evaluation response section (#32166)
Dependencies: Upgrade to joda time 2.10 (#32160)
Remove aliases resolution limitations when security is enabled (#31952)
Ensure that field aliases cannot be used in multi-fields. (#32219)
TESTS: Check for Netty resource leaks (#31861)
...
Currently we check that the queries that QueryStringQueryBuilder#toQuery returns
is one out of a list of many Lucene query classes. This list has extended a lot over time,
since QueryStringQueryBuilder can build all sort of queries. This makes the test hard to
maintain. The recent addition of alias fields which build a BlendedTermQuery show how
easy this test breaks. Also the current assertions doesn't add a lot in terms of catching
errors. This is why we decided to remove this check.
Closes#32234
The parent filter for nested sort should always match **all** parents regardless
of the child queries. It is used to find the boundaries of a single parent and we use
the child query to match all the filters set in the nested tree so there is no need to
repeat the nested filters.
With this change we ensure that we build bitset filters
only to find the root docs (or the docs at the level where the sort applies) that can be reused
among queries.
Closes#31554Closes#32130Closes#31783
Co-authored-by: Dominic Bevacqua <bev@treatwell.com>
* Enhance Parent circuit breaker error message
This adds information about either the current real usage (if tracking "real"
memory usage) or the child breaker usages to the exception message when the
parent circuit breaker trips.
The messages now look like:
```
[parent] Data too large, data for [my_request] would be [211288064/201.5mb], which is larger than the limit of [209715200/200mb], usages [request=157286400/150mb, fielddata=54001664/51.5mb, in_flight_requests=0/0b, accounting=0/0b]
```
Or when tracking real memory usage:
```
[parent] Data too large, data for [request] would be [251/251b], which is larger than the limit of [200/200b], real usage: [181/181b], new bytes reserved: [70/70b]
```
* Only call currentMemoryUsage once by returning structured object
Normally translog operations will not be replayed on the primary.
Following engine is an exception where we replay translog on both
primary and replica as a non-primary strategy. Even though we won't use
the version_type in the following engine, we still need to pass a valid
value for the primary operation in order not to trip assertions in an
engine.
This commit passes version_type EXTERNAL for translog operation if its
origin is primary.
Relates #31945
Resolving wildcards in aliases expression is challenging as we may end
up with no aliases to replace the original expression with, but if we
replace with an empty array that means _all which is quite the opposite.
Now that we support and serialize the original requested aliases,
whenever aliases are replaced we will be able to know what was
initially requested. `MetaData#findAliases` can then be updated to not
return anything in case it gets empty aliases, but the original aliases
were not empty. That means that empty aliases are interpreted as _all
only if they were originally requested that way.
Relates to #31516
* master:
Painless: Simplify Naming in Lookup Package (#32177)
Handle missing values in painless (#32207)
add support for write index resolution when creating/updating documents (#31520)
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
Remove indication of future multi-homing support (#32187)
Rest test - allow for snapshots to take 0 milliseconds
Make x-pack-core generate a pom file
Rest HL client: Add put watch action (#32026)
Build: Remove pom generation for plugin zip files (#32180)
Fix comments causing errors with Java 11
Fix rollup on date fields that don't support epoch_millis (#31890)
Detect and prevent configuration that triggers a Gradle bug (#31912)
[test] port linux package packaging tests (#31943)
Revert "Introduce a Hashing Processor (#31087)" (#32178)
Remove empty @return from JavaDoc
Adjust SSLDriver behavior for JDK11 changes (#32145)
[test] use randomized runner in packaging tests (#32109)
Add support for field aliases. (#32172)
Painless: Fix caching bug and clean up addPainlessClass. (#32142)
Call setReferences() on custom referring tokenfilters in _analyze (#32157)
Fix BwC Tests looking for UUID Pre 6.4 (#32158)
Improve docs for search preferences (#32159)
use before instead of onOrBefore
Add more contexts to painless execute api (#30511)
Add EC2 credential test for repository-s3 (#31918)
A replica can be promoted and started in one cluster state update (#32042)
Fix Java 11 javadoc compile problem
Fix CP for namingConventions when gradle home has spaces (#31914)
Fix `range` queries on `_type` field for singe type indices (#31756)
[DOCS] Update TLS on Docker for 6.3 (#32114)
ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are
Remove versionType from translog (#31945)
Switch distribution to new style Requests (#30595)
Build: Skip jar tests if jar disabled
Painless: Add PainlessClassBuilder (#32141)
Build: Make additional test deps of check (#32015)
Disable C2 from using AVX-512 on JDK 10 (#32138)
Build: Move shadow customizations into common code (#32014)
Painless: Fix Bug with Duplicate PainlessClasses (#32110)
Remove empty @param from Javadoc
Re-disable packaging tests on suse boxes
Docs: Fix missing example script quote (#32010)
[ML] Wait for aliases in multi-node tests (#32086)
[ML] Move analyzer dependencies out of categorization config (#32123)
Ensure to release translog snapshot in primary-replica resync (#32045)
Handle TokenizerFactory TODOs (#32063)
Relax TermVectors API to work with textual fields other than TextFieldType (#31915)
Updates the build to gradle 4.9 (#32087)
Mute :qa:mixed-cluster indices.stats/10_index/Index - all’
Check that client methods match API defined in the REST spec (#31825)
Enable testing in FIPS140 JVM (#31666)
Fix put mappings java API documentation (#31955)
Add exclusion option to `keep_types` token filter (#32012)
[Test] Modify assert statement for ssl handshake (#32072)
Throw an exception for doc['field'].value
if this document is missing a value for the field.
After deprecation changes have been backported to 6.x,
make this a default behaviour in 7.0
Closes#29286
Now write operations like Index, Delete, Update rely on the write-index associated with
an alias to operate against. This means writes will be accepted even when an alias points to multiple indices, so long as one is the write index. Routing values will be used from the AliasMetaData for the alias in the write-index. All read operations are left untouched.
* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)
When building custom tokenfilters without an index in the _analyze endpoint,
we need to ensure that referring filters are correctly built by calling
their #setReferences() method
Fixes#32154
When a replica is fully recovered (i.e., in `POST_RECOVERY` state) we send a request to the master
to start the shard. The master changes the state of the replica and publishes a cluster state to that
effect. In certain cases, that cluster state can be processed on the node hosting the replica
*together* with a cluster state that promotes that, now started, replica to a primary. This can
happen due to cluster state batched processing or if the master died after having committed the
cluster state that starts the shard but before publishing it to the node with the replica. If the master
also held the primary shard, the new master node will remove the primary (as it failed) and will also
immediately promote the replica (thinking it is started).
Sadly our code in IndexShard didn't allow for this which caused [assertions](13917162ad/server/src/main/java/org/elasticsearch/index/seqno/ReplicationTracker.java (L482)) to be tripped in some of our tests runs.
With the introduction of single types in 6.x, the `_type` field is no longer
indexed, which leads to certain queries that were working before throw errors
now. One such query is the `range` query, that, if performed on a single typer
index, currently throws an IAE since the field is not indexed.
This change adds special treatment for this case in the TypeFieldMapper,
comparing the range queries lower and upper bound to the one existing type and
either returns a MatchAllDocs or a MatchNoDocs query.
Relates to #31632Closes#31476
With the introduction of sequence number, we no longer use versionType to
resolve out of order collision in replication and recovery requests.
This PR removes removes the versionType from translog. We can only remove
it in 7.0 because it is still required in a mixed cluster between 6.x and 5.x.
This commit moves additional unit test runners from being dependencies
of the test task to dependencies of check. Without this change,
reproduce lines are incorrect due to the additional test runner not
matching any of the reproduce class/method info.
closes#31964
Previously we create a translog snapshot inside the resync method,
and that snapshot will be closed by the resync listener. However, if
the resync method throws an exception before the resync listener
is initialized, the translog snapshot won't be released.
Closes#32030
Ensure our tests can run in a FIPS JVM
JKS keystores cannot be used in a FIPS JVM as attempting to use one
in order to init a KeyManagerFactory or a TrustManagerFactory is not
allowed.( JKS keystore algorithms for private key encryption are not
FIPS 140 approved)
This commit replaces JKS keystores in our tests with the
corresponding PEM encoded key and certificates both for key and trust
configurations.
Whenever it's not possible to refactor the test, i.e. when we are
testing that we can load a JKS keystore, etc. we attempt to
mute the test when we are running in FIPS 140 JVM. Testing for the
JVM is naive and is based on the name of the security provider as
we would control the testing infrastrtucture and so this would be
reliable enough.
Other cases of tests being muted are the ones that involve custom
TrustStoreManagers or KeyStoreManagers, null TLS Ciphers and the
SAMLAuthneticator class as we cannot sign XML documents in the
way we were doing. SAMLAuthenticator tests in a FIPS JVM can be
reenabled with precomputed and signed SAML messages at a later stage.
IT will be covered in a subsequent PR
The current docs of the put-mapping Java API is currently broken. It its current
form, it creates an index and uses the whole mapping definition given as a JSON
string as the type name. Since we didn't check the index created in the
IndicesDocumentationIT so far this went unnoticed.
This change adds test to catch this error to the documentation test, changes the
documentation so it works correctly now and adds an input validation to
PutMappingRequest#buildFromSimplifiedDef() which was used internally to reject
calls where no mapping definition is given.
Closes#31906
* es/master:
Add Index UUID to `/_stats` Response (#31871)
Painless: Move and Rename Several Methods in the lookup package (#32105)
Bypass highlight query terms extraction on empty fields (#32090)
Switch non-x-pack to new style requests (#32106)
[Rollup] Add new capabilities endpoint for concrete rollup indices (#30401)
Revert "[test] disable packaging tests for suse boxes"
SQL: allow LEFT and RIGHT as function names (#32066)
DOCS: put LIMIT 10 to the SQL query (#32065)
[test] turn on host io cache for opensuse (#32053)
Tweaked Elasticsearch Service links for SEO
Dealing with empty fields in the highlight phase can
slow down the query because the query terms extraction is done independently
on each field. This change shortcuts the highlighting performed by the unified highlighter
for fields that are not present in the document. In such cases there is nothing to higlight so
we don't need to visit the query to build the highligh builder.
* es/master: (21 commits)
Tweaked Elasticsearch Service links for SEO
Watcher: Store username on watch execution (#31873)
Use correct formatting for links (#29460)
Painless: Separate PainlessLookup into PainlessLookup and PainlessLookupBuilder (#32054)
Scripting: Remove dead code from painless module (#32064)
[Rollup] Replace RollupIT with a ESRestTestCase version (#31977)
[TEST] Consistent algorithm usage (#32077)
[Rollup] Fix duplicate field names in test (#32075)
Ensure only parent breaker trips in unit test
Unmute field collapsing rest tests
Fix BWC check after backport
[Tests] Fix failure due to changes exception message (#32036)
Remove unused params from SSource and Walker (#31935)
[Test] Mute MlJobIT#testDeleteJobAfterMissingAliases
Turn off real-mem breaker in REST tests
Turn off real-mem breaker in single node tests
Fix broken OpenLDAP Vagrant QA test
Cleanup Duplication in `PainlessScriptEngine` (#31991)
SCRIPTING: Remove unused MultiSearchTemplateRequestBuilder (#32049)
Fix compile issues introduced by merge (#32058)
...
With this commit we raise the limit of the child circuit breaker used in
the unit test for the circuit breaker service so it is high enough to trip
only the parent circuit breaker. The previous limit was 300 bytes but
theoretically (considering overhead) we could reach 346 bytes. Thus any
value larger than 300 bytes could trip the child circuit breaker leading
to spurious failures.
Relates #31767
* Replace Ingest ScriptContext with Custom Interface
* Make org.elasticsearch.ingest.common.ScriptProcessorTests#testScripting more precise
* Don't mock script factory in ScriptProcessorTests
* Adjust mock script plugin in IT for new API
Make SnapshotInfo and CreateSnapshotResponse parsers lenient for backwards compatibility. Remove extraneous fields from CreateSnapshotRequest toXContent.
* Adds a new auto-interval date histogram
This change adds a new type of histogram aggregation called `auto_date_histogram` where you can specify the target number of buckets you require and it will find an appropriate interval for the returned buckets. The aggregation works by first collecting documents in buckets at second interval, when it has created more than the target number of buckets it merges these buckets into minute interval bucket and continues collecting until it reaches the target number of buckets again. It will keep merging buckets when it exceeds the target until either collection is finished or the highest interval (currently years) is reached. A similar process happens at reduce time.
This aggregation intentionally does not support min_doc_count, offest and extended_bounds to keep the already complex logic from becoming more complex. The aggregation accepts sub-aggregations but will always operate in `breadth_first` mode deferring the computation of sub-aggregations until the final buckets from the shard are known. min_doc_count is effectively hard-coded to zero meaning that we will insert empty buckets where necessary.
Closes#9572
* Adds documentation
* Added sub aggregator test
* Fixes failing docs test
* Brings branch up to date with master changes
* trying to get tests to pass again
* Fixes multiBucketConsumer accounting
* Collects more buckets than needed on shards
This gives us more options at reduce time in terms of how we do the
final merge of the buckeets to produce the final result
* Revert "Collects more buckets than needed on shards"
This reverts commit 993c782d117892af9a3c86a51921cdee630a3ac5.
* Adds ability to merge within a rounding
* Fixes nonn-timezone doc test failure
* Fix time zone tests
* iterates on tests
* Adds test case and documentation changes
Added some notes in the documentation about the intervals that can bbe
returned.
Also added a test case that utilises the merging of conseecutive buckets
* Fixes performance bug
The bug meant that getAppropriate rounding look a huge amount of time
if the range of the data was large but also sparsely populated. In
these situations the rounding would be very low so iterating through
the rounding values from the min key to the max keey look a long time
(~120 seconds in one test).
The solution is to add a rough estimate first which chooses the
rounding based just on the long values of the min and max keeys alone
but selects the rounding one lower than the one it thinks is
appropriate so the accurate method can choose the final rounding taking
into account the fact that intervals are not always fixed length.
Thee commit also adds more tests
* Changes to only do complex reduction on final reduce
* merge latest with master
* correct tests and add a new test case for 10k buckets
* refactor to perform bucket number check in innerBuild
* correctly derive bucket setting, update tests to increase bucket threshold
* fix checkstyle
* address code review comments
* add documentation for default buckets
* fix typo
Because this is a static method on a public API, and one that we encourage
plugin authors to use, the method with the typo is deprecated in 6.x
rather than just renamed.
With this commit we introduce a new circuit-breaking strategy to the parent
circuit breaker. Contrary to the current implementation which only accounts for
memory reserved via child circuit breakers, the new strategy measures real heap
memory usage at the time of reservation. This allows us to be much more
aggressive with the circuit breaker limit so we bump it to 95% by default. The
new strategy is turned on by default and can be controlled with the new cluster
setting `indices.breaker.total.userealmemory`.
Note that we turn it off for all integration tests with an internal test cluster
because it leads to spurious test failures which are of no value (we cannot
fully control heap memory usage in tests). All REST tests, however, will make
use of the real memory circuit breaker.
Relates #31767
* master:
[TEST] Mute SlackMessageTests.testTemplateRender
Docs: Explain closing the high level client
[ML] Re-enable memory limit integration tests (#31328)
[test] disable packaging tests for suse boxes
Add nio transport to security plugin (#31942)
XContentTests : Insert random fields at random positions (#30867)
Force execution of fetch tasks (#31974)
Fix unreachable error condition in AmazonS3Fixture (#32005)
Tests: Fix SearchFieldsIT.testDocValueFields (#31995)
Add Expected Reciprocal Rank metric (#31891)
[ML] Get ForecastRequestStats doc in RestoreModelSnapshotIT (#31973)
SQL: Add support for single parameter text manipulating functions (#31874)
[ML] Ensure immutability of MlMetadata (#31957)
Tests: Mute SearchFieldsIT.testDocValueFields()
muted tests due to #31940
Work around reported problem in eclipse (#31960)
Move build integration tests out of :buildSrc project (#31961)
Tests: Remove use of joda time in some tests (#31922)
[Test] Reactive 3rd party tests on CI (#31919)
SQL: Support for escape sequences (#31884)
SQL: HAVING clause should accept only aggregates (#31872)
Docs: fix typo in datehistogram (#31972)
Switch url repository rest tests to new style requests (#31944)
Switch reindex tests to new style requests (#31941)
Docs: Added note about cloud service to installation and getting started
[DOCS] Removes alternative docker pull example (#31934)
Add Snapshots Status API to High Level Rest Client (#31515)
ingest: date_index_name processor template resolution (#31841)
Test: fix null failure in watcher test (#31968)
Switch test framework to new style requests (#31939)
Switch low level rest tests to new style Requests (#31938)
Switch high level rest tests to new style requests (#31937)
[ML] Mute test failing due to Java 11 date time format parsing bug (#31899)
[TEST] Mute SlackMessageTests.testTemplateRender
Fix assertIngestDocument wrongfully passing (#31913)
Remove unused reference to filePermissionsCache (#31923)
rolling upgrade should use a replica to prevent relocations while running a scroll
HLREST: Bundle the x-pack protocol project (#31904)
Increase logging level for testStressMaybeFlush
Added lenient flag for synonym token filter (#31484)
[X-Pack] Beats centralized management: security role + licensing (#30520)
HLRest: Move xPackInfo() to xPack().info() (#31905)
Docs: add security delete role to api call table (#31907)
[test] port archive distribution packaging tests (#31314)
Watcher: Slack message empty text (#31596)
[ML] Mute failing DetectionRulesIT.testCondition() test
Fix broken NaN check in MovingFunctions#stdDev() (#31888)
Date: Add DateFormatters class that uses java.time (#31856)
[ML] Switch native QA tests to a 3 node cluster (#31757)
Change trappy float comparison (#31889)
Fix building AD URL from domain name (#31849)
Add opaque_id to audit logging (#31878)
re-enable backcompat tests
add support for is_write_index in put-alias body parsing (#31674)
Improve release notes script (#31833)
[DOCS] Fix broken link in painless example
Handle missing values in painless (#30975)
Remove the ability to index or query context suggestions without context (#31007)
Ingest: Enable Templated Fieldnames in Rename (#31690)
[Docs] Fix typo in the Rollup API Quick Reference (#31855)
Ingest: Add ignore_missing option to RemoveProc (#31693)
Add template config for Beat state to X-Pack Monitoring (#31809)
Watcher: Add ssl.trust email account setting (#31684)
Remove link to oss-MSI (#31844)
Painless: Restructure Definition/Whitelist (#31879)
HLREST: Add x-pack-info API (#31870)
Forces fetch tasks to queue even in the event that the queue is
already full. The reasoning is that fetch tasks may only be follow-up
to query tasks, so the number of additional fetch tasks that may enter
the threadpool is expected to be reasonable.
Closes#29442
This test produced different implementations of joda time classes,
depending on if the data was serialized or not (DateTime vs
MutableDateTime). This now uses a common base class to extract the
milliseconds from the data.
Closes#31992
We introduced these changes in #26708 because for CCR.
However, CCR now uses Lucene instead of translog.
This commit reverts these changes so that we can
minimize differences between the ccr and the master branch.
Relates ##26708
Nhat Nguyen
Jim Ferenczi
Alan Woodward
Alexander Reelsen
Daniel Mitterdorfer
Martijn van Groningen
Jason Tedor
Tim Brooks
Armin Braun
Nik Everett
Adrien Grand
Christoph Büscher
Paul Sanwald
Gordon Brown
Sohaib Iftikhar
Julie Tibshirani
tony-dillon
Colin Goodheart-Smithe
Jim Ferenczi
Vladimir Dolzhenko
Vladimir Dolzhenko
Alpar Torok
Lee Hinman
Igor Motov
Simon Willnauer
David Turner
Luca Cavanna
Mark Tozzi
Jonathan Little
Mikita Karaliou
Daniel Mitterdorfer
Mayya Sharipova
Michael Basnight
Ignacio Vera
Andrey Ershov
JB Nizet
Jay Modi
Lee Hinman
Andy Bristol
markharwood
Yannick Welsch
Ryan Ernst
Boaz Leskes
lipsill
Nicholas Knize
Suresh N S
Parth Verma
Shaunak Kashyap
Zachary Tong
Tal Levy
Ioannis Kakavas
javanna
Dimitris Athanasiou
Tim Vernum
Zachary Tong
itsnotv
Jack Conradson
James Baiera