* Updating password usage.
* Add asterisk
* Add explanation of asterisk for settings that are only sometimes required.
* Clarify change password guidance
This adds general overview documentation for data tiers,
the data tiers specific node roles, and their application in
ILM.
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: debadair <debadair@elastic.co>
(cherry picked from commit d588cab74722bfb1d3ca0fea15d10c66af937306)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
#63214 made TypeFieldType a constant field, and fixed things so that it always
emits deprecation warnings whenever it is referenced in a query or aggregation.
However, it also emits warnings when it is used to build a type filter through
the search context; this is unnecessary, as warnings are already emitted by
the REST layer when types are specified as part of the URL, and it is causing
failures in some BWC tests.
This commit adds a specialised typeFilter method to TypeFieldType to handle
this case without emitted any extra warnings. It also removes an unused duplicate
TypeFieldType class that resulted from a backport merge error.
Fixes#63366
As a result of this, we can remove a chunk of code from TypeParsers as well. Tests
for search/index mode analyzers have moved into their own file. This commit also
rationalises the serialization checks for parameters into a single SerializerCheck
interface that takes the values includeDefaults, isConfigured and the value
itself.
Relates to #62988
We were not consistent in checking for node roles before adding listeners.
In some cases we did check the necessity of a CS listener and in others we did not.
This commit fixes a number of cases of redundant listeners that don't apply to all node roles.
In #61906 we agreed on always providing the default value
ShardRouting.UNAVAILABLE_EXPECTED_SHARD_SIZE
when the SnasphotInfoService failed to retrieve the exact
size for a given snapshot shard. The motivation was to
allow the shard allocation to move forward in case of
failures (so that the unassigned shard does not get stuck
in an unassigned state for too long) while relying on the
fallback values for shard sizes.
Sadly a bug in the
SnapshotShardSizeInfo#getShardSize(ShardRouting, long)
makes the default value to be ignored when the snapshot
shard size retrieval previously failed, returning
ShardRouting.UNAVAILABLE_EXPECTED_SHARD_SIZE
instead of the provided default value. With DiskThresholdDecider
also not relying on the provided default value this triggers
some assertion like in #63376 which helped us to spot the bug.
Closes ##63376
The first refreshDiskUsage() refreshes the ClusterInfo update which in turn
calls listeners like DiskThreshMonitor. This one triggers a reroute as
expected and turns an internal checkInProgress flag before submitting
a cluster state update to relocate shards (the internal flag is toggled
again once the cluster state update is processed).
In the test I suspect that the second refreshDiskUsage() may complete
before DiskThreshMonitor's internal flag is set back to its initial state,
resulting in the second ClusterInfo update to be ignored and message
like "[node_t0] skipping monitor as a check is already in progress" to
be logged. Adding another wait for languid events to be processed
before executing the second refreshDiskUsage() should help here.
Closes#62326
This change adds configurable settings to the `CustomRealm` in the QA
project as the correct declaration and use of settings can be a source
of confusion in custom realms.
The "username" "password" and "roles" are now all configurable, which
demonstrates the use of a simple string setting ("username") a secure
setting ("password") and a more complex list setting ("roles").
Backport of: #62287
Strings in the watcher context may use the `.sha1()` and `.sha256()`
augmentation added for ingest.
Ref: #59633, #59671Fixes: #61244
Backport of: 380ee6f
Currently we add translog operation bytes to an array list and flush
them on the next write. Unfortunately, this does not currently play well
with our byte pooling which means each operation is backed, at minimum,
by a 16KB array. This commit improves memory efficiency for small
operations by serializing the operations to an output stream.
This PR updates the `logstash_admin` role to include the recently-added Logstash Pipeline Management APIs, as well as access to the `.logstash*` index pattern.
Co-authored-by: William Brafford <williamrandolphbrafford@gmail.com>
Currently a TranslogWriter add operation is synchronized. This operation
adds the bytes to the file output stream buffer and issues a write
system call if the buffer is filled. This happens every 8KB which means
that we routinely block other add calls on system writes.
This commit modifies the add operation to simply place the operation in
an array list. The array list if flushed when the sync call occurs or
when 1MB is buffered.
The ML cpp notice only exists with default distributions, but the check
task exists on all archive distributions. This commit avoids creating
the task for distributions that don't have ML.
closes#63128
Plugins are loaded in isolated child class loaders of the root class loader. However, some libraries depend on the context class loader being set. This commit sets the context class loader for the duration of calling each plugins constructor.
relates #52320
Co-authored-by: Ryan Ernst <ryan@iernst.net>
When constructing a value fetcher, the 'parsesArrayValue' flag must match
`FieldMapper#parsesArrayValue`. However there is nothing in code or tests to
help enforce this.
This PR reworks the value fetcher constructors so that `parsesArrayValue` is
'false' by default. Just as for `FieldMapper#parsesArrayValue`, field types must
explicitly set it to true and ensure the behavior is covered by tests.
Follow-up to #62974.
An invalid void expression type from a null safe operator caused ClassFormatError for the script Map
x= ['0': 0]; x?.0 > 1. This change sets and propagates the correct expression type for the null safe
operator to be written out.
This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.
Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:
- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`
Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```
The current link points to an obsolete site, which is no longer maintained.
Co-authored-by: Stefan Walter <67258699+rd-stefan-walter@users.noreply.github.com>
Determines the shard size of shards before allocating shards that are
recovering from snapshots. It ensures during shard allocation that the
target node that is selected as recovery target will have enough free
disk space for the recovery event. This applies to regular restores,
CCR bootstrap from remote, as well as mounting searchable snapshots.
The InternalSnapshotInfoService is responsible for fetching snapshot
shard sizes from repositories. It provides a getShardSize() method
to other components of the system that can be used to retrieve the
latest known shard size. If the latest snapshot shard size retrieval
failed, the getShardSize() returns
ShardRouting.UNAVAILABLE_EXPECTED_SHARD_SIZE. While
we'd like a better way to handle such failures, returning this value
allows to keep the existing behavior for now.
Note that this PR does not address an issues (we already have today)
where a replica is being allocated without knowing how much disk
space is being used by the primary.
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
* Cleanup on integtest distribution setup (#62937)
- Simplify build task and archive base name calculation
- Move integ test zip project only setup into integ test zip build script
* Fix merge
* Wire local unreleased bwc versions more efficient for tests (#62473)
For testing against the local distribution we already avoid the packaging/unpackaging
cycle of es distributions when setting up test clusters. This PR adopts the usage of the
expanded created distributions for unreleased bwc versions (versions that are checkout
from a branch and build from source in the :distribution:bwc:minor / :distribution:bwc:bugfix).
This makes the setup of bwc based cross version tests a bit faster by avoiding
the unpackaging overhead. We still assemble both in the bwcBuild tasks atm
which will be addressed in a later issue.
This reworks the :distribution:bwc project:
- Convert all the custom logic from build script logic (groovy) into gradle binary plugins (java)
- Tried to make the bwc setup logic a bit more readable
- Add basic functional test coverage for the bwc logic this PR tweaked.
- Extracted a general internal BWC Git plugin out of the bwc setup plugin to improve maintenance
and testability
- Changed the InternalDistributionPlugin to resolve the extracted distro instead on relying
on unpacking the distribution archive
* Fix java8 incompatibility
* Fix extension calculation for 6.8.* distribution
Add a new ids field to the API of invalidating API keys so that it supports bulk
invalidation with a list of IDs.
Note the existing id field is kept as is and it is an error if both id and ids are specified.
Przemysław Witek
Adam Locke
Luca Cavanna
James Rodewig
Lisa Cawley
Andrei Dan
Alan Woodward
Mayya Sharipova
Przemyslaw Gomulka
Armin Braun
Tanguy Leroux
David Kyle
Hendrik Muhs
Tim Vernum
Stuart Tettemer
Tim Brooks
Gordon Brown
Ryan Ernst
Dawid Weiss
Julie Tibshirani
Jack Conradson
Rene Groeschke
Lee Hinman
Igor Motov
István Zoltán Szabó
Dan Hermann
Yang Wang