This commit adds documentation for configuring Kerberos realm.
Configuring Kerberos realm documentation highlights important
terminology and requirements before creating Kerberos realm.
Most of the documentation is centered around configuration from
Elasticsearch rather than go deep into Kerberos implementation.
Kerberos realm settings are mentioned in the security settings
for Kerberos realm.
When a list/an array of cron expressions is provided, and one of those addresses
is already expired, the expired one will be considered as an option
instead of the valid next one.
This commit also reduces the visibility of the CronnableSchedule and
refactors a comparator to look like java 8.
This is a followup to #31886. After that commit the
TransportConnectionListener had to be propogated to both the
Transport and the ConnectionManager. This commit moves that listener
to completely live in the ConnectionManager. The request and response
related methods are moved to a TransportMessageListener. That listener
continues to live in the Transport class.
* elastic/master: (46 commits)
NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS (#32764)
[DOCS] Splits the users API documentation into multiple pages (#32825)
[DOCS] Splits the token APIs into separate pages (#32865)
[DOCS] Creates redirects for role management APIs page
Bypassing failing test PainlessDomainSplitIT#testHRDSplit (#32966)
TEST: Mute testRetentionPolicyChangeDuringRecovery
[DOCS] Fixes more broken links to role management APIs
[Docs] Tweaks and fixes to rollup docs
[DOCS] Fixes links to role management APIs
[ML][TEST] Fix BasicRenormalizationIT after adding multibucket feature
[DOCS] Splits the roles API documentation into multiple pages (#32794)
[TEST] Run pre 6.4 nodes in non-FIPS JVMs (#32901)
Make Geo Context Mapping Parsing More Strict (#32821)
[ML] fix updating opened jobs scheduled events (#31651) (#32881)
Scripted metric aggregations: add deprecation warning and system property to control legacy params (#31597)
Tests: Fix timezone conversion in DateTimeUnitTests
Enable FIPS140LicenseBootstrapCheck (#32903)
Fix InternalAutoDateHistogram reproducible failure (#32723)
Remove assertion in testDocStats on deletedDocs counter (#32914)
HLRC: Move ML request converters into their own class (#32906)
...
* Lazy resolve DNS (i.e. `String` to `DiscoveryNode`) to not run into indefinitely caching lookup issues (provided the JVM dns cache is configured correctly as explained in https://www.elastic.co/guide/en/elasticsearch/reference/6.3/networkaddress-cache-ttl.html)
* Changed `InetAddress` type to `String` for that higher up the stack
* Passed down `Supplier<DiscoveryNode>` instead of outright `DiscoveryNode` from `RemoteClusterAware#buildRemoteClustersSeeds` on to lazy resolve DNS when the `DiscoveryNode` is actually used (could've also passed down the value of `clusterName = REMOTE_CLUSTERS_SEEDS.getNamespace(concreteSetting)` together with the `List<String>` of hosts, but this route seemed to introduce less duplication and resulted in a significantly smaller changeset).
* Closes#28858
* Store phase steps for index in PolicyStepsRegistry
This changes the way that steps are retrieved from `PolicyStepsRegistry` to
store the steps on a per-index basis (in memory for now, though that will change
in subsequent PRs). These steps are rebuilt as the index changes phases.
This also fixes a bug where an action with the same phase and name was not being
considered changed (and thus updated) in the compiled steps list. These are now
correctly considered as "upsert" diffs.
Relates to #29823
- Missing links to new IndexCaps API
- Incorrect security permissions on IndexCaps API
- GetJobs API must supply a job (or `_all`), omitting throws error
- Link to search/agg limitations from RollupSearch API
- Tweak URLs in quick reference
- Formatting of overview page
As the multibucket feature was merged in, this test hit a side effect
which means buckets trailing an anomaly could become anomalous.
This commit fixes the problem by filtering low score records when
we request them.
Elasticsearch versions earlier than 6.4.0 cannot properly run in a
FIPS 140 JVM. This commit ensures that we use a non-FIPS JVM for
nodes that we spin up in BWC tests even when we're testing FIPS.
* ML: fix updating opened jobs scheduled events (#31651)
* Adding UpdateParamsTests license header
* Adding integration test and addressing PR comments
* addressing test and job names
This commit removes the put privilege API in favor of having a single API to
create and update privileges. If we see the need to have an API like this in
the future we can always add it back.
The Kibana settings docs that these watches rely on can sometimes
contain no xpack settings. When this is the case, we will end up with a
null pointer exception in the script. We need to guard against in these
scripts so this commit does that.
this PR creates client-side instances of the following actions:
- DeleteAction
- ForceMergeAction
- ReadOnlyAction
- RolloverAction
- ForceMergeAction
AllocateAction was done separately
As we migrate to a per-phase execution model, we need to prepare our cluster-state-step execution model to be aligned. It is the case that the final iteration into the next "currentStep" from the next phase would not be available in the registry yet. This change exits the execution loop early as to not jump into executing the next phase's steps before the registry is properly updated
This change cleans up some methods in the CharArrays class from x-pack, which
includes the unification of char[] to utf8 and utf8 to char[] conversions that
intentionally do not use strings. There was previously an implementation in
x-pack and in the reloading of secure settings. The method from the reloading
of secure settings was adopted as it handled more scenarios related to the
backing byte and char buffers that were used to perform the conversions. The
cleaned up class is moved into libs/core to allow it to be used by requests
that will be migrated to the high level rest client.
Relates #32332
* master:
Fix global checkpoint listeners test
HLRC: adding machine learning open job (#32860)
[ML] Add log structure finder functionality (#32788)
INGEST: Add Configuration Except. Data to Metdata (#32322)
This change adds a library to ML that can be used to deduce a log
file's structure given only a sample of the log file.
Eventually this will be used to add an endpoint to ML to make the
functionality available to end users, but this will follow in a
separate change.
The functionality is split into a library so that it can also be
used by a command line tool without requiring the command line
tool to include all server code.
This removes custom Response classes that extend `AcknowledgedResponse` and do nothing, these classes are not needed and we can directly use the non-abstract super-class instead.
While this appears to be a large PR, no code has actually changed, only class names have been changed and entire classes removed.
[ML] Removing old per-partition normalization code
Per-partition normalization is an old, undocumented feature that was
never used by clients. It has been superseded by per-partition maximum
scoring.
To maintain communication compatibility with nodes prior to 6.5 it is
necessary to maintain/cope with the old wire format
It is useful to have a random TimeseriesLifecycleType-backed LifecyclePolicy
for testing. This PR exposes a helper method to create one and use it for serialization tests
in LifecyclePolicyTests
This change removes the PasswordHashingBootstrapCheck and replaces it
with validation on the setting itself. This ensures we always get a
valid value from the setting when it is used.
This change moves the validation for values of usernames and passwords
from the request to the transport action. This is done to prevent
the need to move more classes into protocol once we add this API to the
high level rest client. Additionally, this resolves an issue where
validation depends on settings and we always pass empty settings
instead of the actual settings.
Relates #32332
The HipChatMessage#render is no longer used, and instead the
HipChatAccount#render is used in the ExecutableHipChatAction. Only a
test that validated the HttpProxy used this render method still. This
commit cleans it up.
The auth.basic package was an example of a single implementation
interface that leaked into many different classes. In order to clean
this up, the HttpAuth interface, factories, and Registries all were
removed and the single implementation, BasicAuth, was substituted in all
cases. This removes some dependenies between Auth and the Templates,
which can now use static methods on BasicAuth. BasicAuth was also moved
into the http package and all of the other classes were removed.
The PagerDuty v1 API is EOL and will stop accepting new accounts
shortly. This commit swaps out the watcher use of the v1 API with the
new v2 API. It does not change anything about the existing watcher
API.
Closes#32243
All Unit tests in this module are muted in FIPS 140 JVMs and
as such the CI run fails. This commit disables test task for the
module in a FIPS JVM and reverts adding a dummy test in
4cbcc1.
We only upgrade the ID when the state is saved in one of four scenarios:
- when we reach a checkpoint (every 50 pages)
- when we run out of data
- when explicitly stopped
- on failure
The test was relying on the pre-upgrade to finish, save state and then
the post-upgrade to start, hit the end of data and upgrade ID. THEN
get the new doc and apply the new ID.
But I think this is vulnerable to timing issues. If the pre-upgrade
portion shutdown before it saved the state, when restarting we would run
through all the data from the beginning with the old ID, meaning both
docs would still have the old scheme.
This change makes the pre-upgrade wait for the job to go back to STARTED
so that we know it persisted the end point. Post-upgrade, it stops and
restarts the job to ensure the state was persisted and the ID upgraded.
That _should_ rule out the above timing issue.
Closes#32773
Added infrastructure to push through the 'person name field value' to
the normalizer process. This is required by the normalizer to retrieve
the maximum scores for individual partitions.
The request and response classes have been extracted from `IndexUpgradeInfoAction` into top-level classes, and moved to the protocol jar. The `UpgradeActionRequired` enum is also moved.
Relates to #29827
This commit adds missing debug log statements for exceptions
that occur during ticket validation. I thought these
get logged somewhere else in authentication chain
but even after enabling trace logs I could not see them
logged. As the Kerberos exception messages are cryptic
adding full stack trace would help debugging faster.
* Clear Job#finished_time when it is opened (#32605)
* not returning failure when Job#finished_time is not reset
* Changing error log string and source string
Our rest testing framework has support for sniffing the host metadata on
startup and, before this change, it'd sniff that metadata before running
the first test. This prevents running these tests against
elasticsearch installations that won't support sniffing like Elastic
Cloud. This change allows tests to only sniff for metadata when they
encounter a test with a `node_selector`. These selectors are the things
that need the metadata anyway and they are super rare. Tests that use
these won't be able to run against installations that don't support
sniffing but we can just skip them. In the case of Elastic Cloud, these
tests were never going to work against Elastic Cloud anyway.
The upcoming ML log structure finder functionality will use these
libraries, and it makes sense to use the same versions that are
being used elsewhere in Elasticsearch. This is especially true
with icu4j, which is pretty big.
This commit modifies the test to handle file permission
tests in windows/dos environments. The test requires access
to UserPrincipal and so have modified the plugin-security policy
to access user information.
Closes#32637
The qa tests with security haven't actually gone as far as testing security roles yet, so this is a start in the hopes of both bringing the tests into the ilm plugin
* Adds REST client support for PutOperationMode in ILM
* Corrects licence headers
* iter
* add request converter test
* Fixes tests
* Creates start and stop actions for controlling ILM operation
* Addresses review comments
The incorrect NodeInfo is created when the optional parameter is not used, leading to the incorrect constructor being used. Simplified LocateFunctionProcessorDefinition by using one constructor instead of two.
Fixes https://github.com/elastic/elasticsearch/issues/32554
Skip the comparative tests using lowercasing/uppercasing against H2 (which considers the Locale).
ES-SQL is, so far, ignoring the Locale.
Still, the same queries are executed against ES-SQL alone and results asserted to be correct.
We previously discussed moving the classes extending `AcknowledgedResponse` to
simply use `AcknowledgedResponse`, making the class non-abstract.
This moves the first class to do this, removing `WritePipelineResponse` in the
process.
If we like the way this looks, I will switch the remaining classes over to using
`AcknowledgedResponse`.
Since replica counts and allocation rules are set separately, it is not always clear how many replicas are to be allocated in the allocate action. Moving the replicas action to occur at the same time as the allocate action, resolves this confusion that could end an undesired state. This means that the ReplicasAction is removed, and a new optional replicas parameter is added to AllocateAction.
This was originally set to a few seconds while prototyping things.
This interval is for the scheduled trigger of policies. Policies
have this extra trigger beyond just on cluster-state changes because
cluster-state changes may not be happeneing in a cluster for
whatever reason, and we need to continue making progress. Updating
this value to be larger is reasonable since not all operations
are expected to be completed in the span of seconds, but instead in
minutes and hours. 10 minutes is sane.
* Remove UpdateSettingsTestHelper class
By making the `settings()` method public on `UpdateSettingsRequest` (I think it
should have been in the first place) we can get rid of this class entirely. Mock
response objects are now constructed by parsing JSON without making the
constructor public.
Relates to #29823
* Remove RolloverIndexTestHelper
This removes the `RolloverIndexTestHelper` class in favor of making a couple of
getters publically accessible as well as custom building a response object using
JSON parsing.
Relates to #29823
This commit removes the hacks associated with mocking Response objects. Rather
than parse a wrapped byte array, the constructors for `IndicesAliasesResponse`
and `ResizeResponse` are made public
Relates to #29823
* master:
Cross-cluster search: preserve cluster alias in shard failures (#32608)
Handle AlreadyClosedException when bumping primary term
[TEST] Allow to run in FIPS JVM (#32607)
[Test] Add ckb to the list of unsupported languages (#32611)
SCRIPTING: Move Aggregation Scripts to their own context (#32068)
Painless: Use LocalMethod Map For Lookup at Runtime (#32599)
[TEST] Enhance failure message when bulk updates have failures
[ML] Add ML result classes to protocol library (#32587)
Suppress LicensingDocumentationIT.testPutLicense in release builds (#32613)
[Rollup] Update wire version check after backport
Suppress Wildfly test in FIPS JVMs (#32543)
[Rollup] Improve ID scheme for rollup documents (#32558)
ingest: doc: move Dot Expander Processor doc to correct position (#31743)
[ML] Add some ML config classes to protocol library (#32502)
[TEST]Split transport verification mode none tests (#32488)
Core: Move helper date formatters over to java time (#32504)
[Rollup] Remove builders from DateHistogramGroupConfig (#32555)
[TEST} unmutes SearchAsyncActionTests and adds debugging info
[ML] Add Detector config classes to protocol library (#32495)
[Rollup] Remove builders from MetricConfig (#32536)
Tests: Add rolling upgrade tests for watcher (#32428)
Fix race between replica reset and primary promotion (#32442)
Rest HL client: Add get license action
Continues to use String instead of a more complex License class to
hold the license text similarly to put license.
Relates #29827
The Apache Http components support for Spnego scheme
uses canonical name by default.
Also when resolving host name, on centos by default
there are other aliases so adding them to the
DelegationPermission.
Closes#32498
If a leader index is deleted while there is an active follower, the
follower will send shard changes requests bound for the leader
index. Today this will result in a null pointer exception because there
will not be an index routing table for the index. A null pointer
exception looks like a bug to a user so this commit addresses this by
throwing an index not found exception instead.
* Change SecurityNioHttpServerTransportTests to use PEM key and
certificate files instead of a JKS keystore so that this tests
can also run in a FIPS 140 JVM
* Do not attempt to run cases with ssl.verification_mode NONE in
SessionFactoryTests so that the tests can run in a FIPS 140 JVM
This commit addresses a race that can happen in the basic CCR stats REST
tests. Namely, peek reads can fire before the REST test client fires the
stats request. This means that we have to weaken our assertions about
the expected stats response.
This commit adds the ML results classes to the X-Pack protocol
library used by the high level REST client.
(Other commits will add the config classes and stats classes.)
These classes:
- Are publically immutable
- Are privately mutable - this is perhaps not as nice as the
config classes, but to do otherwise would require adding
builders and the corresponding server-side classes that the
old transport client used don't have builders
- Have little/no validation of field values beyond null checks
- Are convertible to and from X-Content, but NOT wire transportable
- Have lenient parsers to maximize compatibility across versions
- Have the same class names and getter names as the corresponding
classes in X-Pack core to ease migration for transport client
users
- Don't reproduce all the methods that do calculations or
transformations that the the corresponding classes in X-Pack core
have
Bumping down the version to 6.4 since the backport is complete. Also
adds some missing version checks to the bwc tests to make sure it
only runs on the correct versions
Previously, we were using a simple CRC32 for the IDs of rollup documents.
This is a very poor choice however, since 32bit IDs leads to collisions
between documents very quickly.
This commit moves Rollups over to a 128bit ID. The ID is a concatenation
of all the keys in the document (similar to the rolling CRC before),
hashed with 128bit Murmur3, then base64 encoded. Finally, the job
ID and a delimiter (`$`) are prepended to the ID.
This gurantees that there are 128bits per-job. 128bits should
essentially remove all chances of collisions, and the prepended
job ID means that _if_ there is a collision, it stays "within"
the job.
BWC notes:
We can only upgrade the ID scheme after we know there has been a good
checkpoint during indexing. We don't rely on a STARTED/STOPPED
status since we can't guarantee that resulted from a real checkpoint,
or other state. So we only upgrade the ID after we have reached
a checkpoint state during an active index run, and only after the
checkpoint has been confirmed.
Once a job has been upgraded and checkpointed, the version increments
and the new ID is used in the future. All new jobs use the
new ID from the start
This commit adds four ML config classes to the X-Pack protocol
library used by the high level REST client.
(Other commits will add the remaining config classes, plus results
and stats classes.)
These classes:
- Are immutable
- Have little/no validation of field values beyond null checks
- Are convertible to and from X-Content, but NOT wire transportable
- Have lenient parsers to maximize compatibility across versions
- Have the same class names, member names and getter/setter names
as the corresponding classes in X-Pack core to ease migration
for transport client users
- Don't reproduce all the methods that do calculations or
transformations that the the corresponding classes in X-Pack core
have
This commit splits SecurityNetty4TransportTests in two methods
one handling verification mode certificate and full and one
handling verification mode none. This is done so that the second
method can be muted in a FIPS 140 JVM where verification mode none
cannot be used.
Same motivation as #32507 but for the DateHistogramGroupConfig
configuration object. This pull request also changes the format of the
time zone from a Joda's DateTimeZone to a simple String.
It should help to port the API to the high level rest client and allows
clients to not be forced to use the Joda Time library. Serialization is
impacted but does not need a backward compatibility layer as
DateTimeZone are serialized as String anyway. XContent also expects
a String for timezone, so I found it easier to move everything to String.
Related to #29827
This commit adds the Detector class and its dependencies to the
X-Pack protocol library used by the high level REST client.
(Future commits will add the remaining config classes, plus results
and stats classes.)
These classes:
- Are immutable, with builders, but the builders do no validation
beyond null checks
- Are convertible to and from X-Content, but NOT wire transportable
- Have lenient parsers to maximize compatibility across versions
- Have the same class names, member names and getter/setter names
as the corresponding classes in X-Pack core to ease migration
for transport client users
- Don't reproduce all the methods that do calculations or
transformations that the the corresponding classes in X-Pack core
have
These tests ensure, that the basic watch APIs are tested in the rolling
upgrade tests. After initially adding a watch, the tests try to get,
execute, deactivate and activate a watch. Watcher stats are tested as
well, and an own java based test has been added for restarting, as that
requires waiting for a state change. Watcher history is also checked.
Closes#31216
* master:
HLRC: Move commercial clients from XPackClient (#32596)
Add cluster UUID to Cluster Stats API response (#32206)
Security: move User to protocol project (#32367)
[TEST] Test for shard failures, add debug to testProfileMatchesRegular
Minor fix for javadoc (applicable for java 11). (#32573)
Painless: Move Some Lookup Logic to PainlessLookup (#32565)
TEST: Avoid merges in testSeqNoAndCheckpoints
[Rollup] Remove builders from HistoGroupConfig (#32533)
Mutes failing SQL string function tests due to #32589
fixed elements in array of produced terms (#32519)
INGEST: Enable default pipelines (#32286)
Remove cluster state initial customs (#32501)
Mutes LicensingDocumentationIT due to #32580
[ML] Remove multiple_bucket_spans (#32496)
[ML] Rename JobProvider to JobResultsProvider (#32551)
Correct minor typo in explain.asciidoc for HLRC
Build: Add elastic maven to repos used by BuildPlugin (#32549)
Clarify the error message when a pipeline agg is used in the 'order' parameter. (#32522)
Revert "[test] turn on host io cache for opensuse (#32053)"
Enable packaging tests on suse boxes
[ML] Improve error when no available field exists for rule scope (#32550)
[ML] Improve error for functions with limited rule condition support (#32548)
Painless: Clean Up PainlessField (#32525)
Add @AwaitsFix for #32554
Remove broken @link in Javadoc
Scripting: Conditionally use java time api in scripting (#31441)
[ML] Fix thread leak when waiting for job flush (#32196) (#32541)
Add AwaitsFix to failing test - see #32546
Core: Minor size reduction for AbstractComponent (#32509)
SQL: Added support for string manipulating functions with more than one parameter (#32356)
[DOCS] Reloadable Secure Settings (#31713)
Watcher: Reenable HttpSecretsIntegrationTests#testWebhookAction test (#32456)
[Rollup] Remove builders from TermsGroupConfig (#32507)
Use hostname instead of IP with SPNEGO test (#32514)
Switch x-pack rolling restart to new style Requests (#32339)
NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
[DOCS] Small fixes in rule configuration page (#32516)
Painless: Clean up PainlessMethod (#32476)
Build: Remove shadowing from benchmarks (#32475)
Docs: Add all JDKs to CONTRIBUTING.md
Add licensing enforcement for FIPS mode (#32437)
SQL: Add test for handling of partial results (#32474)
Mute testFilterCacheStats
[ML][DOCS] Fix typo applied_to => applies_to
Scripting: Fix painless compiler loader to know about context classes (#32385)
For a new feature like CCR we will go without this extra layer of
indirection. This commit replaces all /_xpack/ccr/_(\S+) endpoints by
/_ccr/$1 endpoints.
* Make cluster stats response contain cluster UUID
* Updating constructor usage in Monitoring tests
* Adding cluster_uuid field to Cluster Stats API reference doc
* Adding rest api spec test for expecting cluster_uuid in cluster stats response
* Adding missing newline
* Indenting do section properly
* Missed a spot!
* Fixing the test cluster ID
The User class has been moved to the protocol project for upcoming work
to add more security APIs to the high level rest client. As part of
this change, the toString method no longer uses a custom output method
from MetadataUtils and instead just relies on Java's toString
implementation.
This commit does the following:
- renames index-lifecycle plugin to ilm
- modifies the endpoints to ilm instead of index_lifecycle
- drops _xpack from the endpoints
- drops a few duplicate endpoints
This commit makes the `index.lifecycle.name` setting internal an index, this
means that the policy can only be set on the index creation, or with the
specialized `RestSetIndexLifecyclePolicy` action.
Relates to #29823
This commit removes the never released multiple_bucket_spans
configuration parameter. This is now replaced with the new
multibucket feature that requires no configuration.
Added support for string manipulating functions with more than one parameter:
CONCAT, LEFT, RIGHT, REPEAT, POSITION, LOCATE, REPLACE, SUBSTRING, INSERT
The error message mentioned in #30094 does not link to to a cause by the
test itself, as there are still inflight requests according to the
circuit breaker.
I ran this test class 100k times on bare metal and could not reproduce
it. I will reenable the test for now.
Closes#30094
While working on adding the Create Rollup Job API to the
high level REST client (#29827), I noticed that the configuration
objects like TermsGroupConfig rely on the Builder pattern in
order to create or parse instances. These builders are doing
some validation but the same validation could be done within
the constructor itself or on the server side when appropriate.
This commit removes the builder for TermsGroupConfig,
removes some other methods that I consider not really usefull
once the TermsGroupConfig object will be exposed in the
high level REST client. It also simplifies the parsing logic.
Related to #29827
This PR re-introduces our ILM integration tests with mock steps
that we can control in the tests.
These tests uncovered a bug where the policy-steps-registry was
not being updated on newly elected masters when there were no
cluster-state changes to ILM metadata. The fix layed out cleans up
the registry/runner when a node is un-elected as master. It re-assigns
the class variables so that the existing runner/registry instances that
may be running can continue to do so in other threads, potentially.
This change updates KerberosAuthenticationIT to resolve the host used
to connect to the test cluster. This is needed because the host could
be an IP address but SPNEGO requires a hostname to work properly. This
is done by adding a hook in ESRestTestCase for building the HttpHost
from the host and port.
Additionally, the project now specifies the IPv4 loopback address as
the http host. This is done because we need to be able to resolve the
address used for the HTTP transport before the node starts up, but the
http.ports file is not written until the node is started.
Closes#32498
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:qa:rolling-upgrade*` projects to use
the new versions.
* Upgrade to `4.1.28` since the problem reported in #32487 is a bug in Netty itself (see https://github.com/netty/netty/issues/7337)
* Fixed other leaks in test code that now showed up due to fixes improvements in leak reporting in the newer version
* Needed to extend permissions for netty common package because it now sets a classloader at runtime after changes in 63bae0956a
* Adjusted forbidden APIs check accordingly
* Closes#32487
This commit adds licensing enforcement for FIPS mode through the use of
a bootstrap check, a node join validator, and a check in the license
service. The work done here is based on the current implementation of
the TLS enforcement with a production license.
The bootstrap check is always enforced since we need to enforce the
licensing and this is the best option to do so at the present time.
First, some background: we have 15 different methods to get a logger in
Elasticsearch but they can be broken down into three broad categories
based on what information is provided when building the logger.
Just a class like:
```
private static final Logger logger = ESLoggerFactory.getLogger(ActionModule.class);
```
or:
```
protected final Logger logger = Loggers.getLogger(getClass());
```
The class and settings:
```
this.logger = Loggers.getLogger(getClass(), settings);
```
Or more information like:
```
Loggers.getLogger("index.store.deletes", settings, shardId)
```
The goal of the "class and settings" variant is to attach the node name
to the logger. Because we don't always have the settings available, we
often use the "just a class" variant and get loggers without node names
attached. There isn't any real consistency here. Some loggers get the
node name because it is convenient and some do not.
This change makes the node name available to all loggers all the time.
Almost. There are some caveats are testing that I'll get to. But in
*production* code the node name is node available to all loggers. This
means we can stop using the "class and settings" variants to fetch
loggers which was the real goal here, but a pleasant side effect is that
the ndoe name is now consitent on every log line and optional by editing
the logging pattern. This is all powered by setting the node name
statically on a logging formatter very early in initialization.
Now to tests: tests can't set the node name statically because
subclasses of `ESIntegTestCase` run many nodes in the same jvm, even in
the same class loader. Also, lots of tests don't run with a real node so
they don't *have* a node name at all. To support multiple nodes in the
same JVM tests suss out the node name from the thread name which works
surprisingly well and easy to test in a nice way. For those threads
that are not part of an `ESIntegTestCase` node we stick whatever useful
information we can get form the thread name in the place of the node
name. This allows us to keep the logger format consistent.
This commit adds an assumption to two test methods in
SSLTrustRestrictionsTests that we are not on JDK 11 as the tests
currently fail there.
Relates #29989
Since the reason for a step not being found in a registry may be due to staleness of the
registry between it and the cluster state, we do not want to throw an IllegalStateException.
Staleness is something that will be self-healing after follow-up applications of the cluster state
updates, so this is a recoverable issue that should log a warning instead of throwing an exception
Closes#32181.
This commit removes Kerberos bootstrap checks as they were more
validation checks and better done in Kerberos realm constructor
than as bootstrap checks. This also moves the check
for one Kerberos realm per node to where we initialize realms.
This commit adds few validations which were missing earlier
like missing read permissions on keytab file or if it is directory
to throw exception with error message.
Today ShardFollowNodeTask might fetch some operations more than once.
This happens because we ask the leading for up to max_batch_count
operations (instead of the left-over size) for the left-over request.
The leading then can freely respond up to the max_batch_count, and at
the same time, if one of the previous requests completed, we might issue
another read request whose range overlaps with the response of the
left-over request.
Closes#32453
The default behaviour for "GetPrivileges" is to get all application
privileges. This should only be allowed if the user has access to
the "*" application.
This adds HLRC support for the ILM operation of setting an index's lifecycle
policy.
It also includes extracting and renaming a number of classes (like the request
and response objects) as well as the addition of a new `IndexLifecycleClient`
for the HLRC. This is a prerequisite to making the `index.lifecycle.name`
setting internal only, because we require a dedicated REST endpoint to change
the policy, and our tests currently set this setting with the REST client
multiple places. A subsequent PR will change the setting to be internal and move
those uses over to this new API.
This misses some links to the documentation because I don't think ILM has any
documentation available yet.
Relates to #29827 and #29823
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/plugin/security` project to use the new
versions.
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/qa/security-example-spi-extension`
project to use the new versions.
These are only ever set internally during regular ILM execution, they don't need
to be set otherwise.
A subsequent PR will work on adding a dedicated endpoint for the
`LIFECYCLE_NAME` setting so it can be changed by a user (and then marked as
`InternalIndex` as well)
Relates to #29823
* master:
Tests: Fix convert error tests to use fixed value (#32415)
IndicesClusterStateService should replace an init. replica with an init. primary with the same aId (#32374)
REST high-level client: parse back _ignored meta field (#32362)
[CI] Mute DocumentSubsetReaderTests testSearch
Today we do not check if the `following_index` setting of the follower
is enabled or not when processing a follow-request. If that setting is
disabled, the follower will use the default engine, not the following
engine. This change checks and rejects such invalid follow requests.
Relates #30086
* master:
Remove reference to non-existent store type (#32418)
[TEST] Mute failing FlushIT test
Fix ordering of bootstrap checks in docs (#32417)
[TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints
[TEST] Mute failing testConvertLongHexError
bump lucene version after backport
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
[Kerberos] Avoid vagrant update on precommit (#32416)
TESTS: Move netty leak detection to paranoid level (#32354)
[DOCS] Fixes formatting of scope object in job resource
Copy missing segment attributes in getSegmentInfo (#32396)
AbstractQueryTestCase should run without type less often (#28936)
INGEST: Fix Deprecation Warning in Script Proc. (#32407)
Switch x-pack/plugin to new style Requests (#32327)
Docs: Correcting a typo in tophits (#32359)
Build: Stop double generating buildSrc pom (#32408)
TEST: Avoid triggering merges in FlushIT
Fix missing JavaDoc for @throws in several places in KerberosTicketValidator.
Switch x-pack full restart to new style Requests (#32294)
Release requests in cors handler (#32364)
Painless: Clean Up PainlessClass Variables (#32380)
Docs: Fix callouts in put license HL REST docs (#32363)
[ML] Consistent pattern for strict/lenient parser names (#32399)
Update update-settings.asciidoc (#31378)
Remove some dead code (#31993)
Introduce index store plugins (#32375)
Rank-Eval: Reduce scope of an unchecked supression
Make sure _forcemerge respects `max_num_segments`. (#32291)
TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests (#32377)
Only enforce password hashing check if FIPS enabled (#32383)
if policy update on index means current step no longer exists
This change only updates the setPolicy for index to add this
functionality. The update policy API will be changed in a follow up PR.
Today it's possible to encounter an Index operation in Lucene whose
_source is disabled, and _recovery_source was pruned by the MergePolicy.
If it's the case, we create a Translog#Index without source and let the
caller validate it later. However, this approach is challenging for the
caller.
Deletes and No-Ops don't allow invoking "source()" method. The caller
has to make sure to call "source()" only on index operations. The
current implementation in CCR does not follow this and fail to replica
deletes or no-ops. Moreover, it's easier to reason if a Translog#Index
always has the source.
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
This commit avoids dependency during compile on copy keytab to
be present in the generated sources so pre-commit does not
stall for updating vagrant box.
Closes#32387
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/plugin` project to use the new versions.
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:qa:full-cluster-restart` project to use
the new versions.
Previously we had two patterns for naming of strict
and lenient parsers.
Some classes had CONFIG_PARSER and METADATA_PARSER,
and used an enum to pass the parser type to nested
parsers.
Other classes had STRICT_PARSER and LENIENT_PARSER
and used ternary operators to pass the parser type
to nested parsers.
This change makes all ML classes use the second of
the patterns described above.
Removing some dead code or supressing warnings where apropriate. Most of the
time the variable tested for null is dereferenced earlier or never used before.
Yogesh Gaikwad
Alexander Reelsen
Alpar Torok
Tim Brooks
Jason Tedor
Armin Braun
Lisa Cawley
Lee Hinman
Tal Levy
Benjamin Trent
Zachary Tong
Dimitris Athanasiou
Ioannis Kakavas
Jay Modi
Ed Savage
Ed Savage
Adrien Grand
David Kyle
Jim Ferenczi
debadair
David Roberts
Nhat Nguyen
Michael Basnight
Colin Goodheart-Smithe
Yannick Welsch
Zachary Tong
Luca Cavanna
Nik Everett
Albert Zaharovits
Tim Vernum
Andrei Stefan
Nicholas Knize
Jim Ferenczi
Tim Vernum
Tanguy Leroux
Albert Zaharovits
Igor Motov
Lee Hinman
Shaunak Kashyap
David Turner
Daniel Mitterdorfer
Jack Conradson
Christoph Büscher