* Retain the expiryDate for trial licenses
While updating the license signature to the new license spec retain
the trial license expiration date to that of the existing license.
Resolves#30882
Today before reading operations on the leading shard, we minimization
the requesting range with the global checkpoint. However, this might
make the request invalid if the following shard generates a requesting
range based on the global-checkpoint from a primary shard and sends
that request to a replica whose global checkpoint is lagged.
Another issue is that we are mutating the request when applying
minimization. If the request becomes invalid on a replica, we will
reroute the mutated request instead of the original one to the primary.
This commit removes the minimization and replaces it by a range check
with the local checkpoint.
* Ensure that a purposefully wrong key is used
Uses a specific keypair for tests that require a purposefully wrong
keypair instead of selecting one randomly from the same pull from
which the correct one is selected. Entropy is low because of the
small space and the same key can be randomly selected as both the
correct one and the wrong one, causing the tests to fail.
The purposefully wrong key is also used in
testSigningKeyIsReloadedForEachRequest and needs to be cleaned
up afterwards so the rest of the tests don't use that for signing.
Resolves#30970
This commit removes the RequestBuilder generic type from Action. It was
needed to be used by the newRequest method, which in turn was used by
client.prepareExecute. Both of these methods are now removed, along with
the existing users of prepareExecute constructing the appropriate
builder directly.
This commit reworks the Sniffer component to simplify it and make it possible to test it.
In particular, it no longer takes out the host that failed when sniffing on failure, but rather relies on whatever the cluster returns. This is the result of some valid comments from #27985. Taking out one single host is too naive, hard to test and debug.
A new Scheduler abstraction is introduced to abstract the tasks scheduling away and make it possible to plug in any test implementation and take out timing aspects when testing.
Concurrency aspects have also been improved, synchronized methods are no longer required. At the same time, we were able to take #27697 and #25701 into account and fix them, especially now that we can more easily add tests.
Last but not least, unit tests are added for the Sniffer component, long overdue.
Closes#27697Closes#25701
* master:
[CI] Mute Ml rolling upgrade tests
Fix license on AcitveDirectorySIDUtil (#30972)
[Test] Prefer ArrayList over Vector (#30965)
[CI] Mute HttpSecretsIntegrationTests#testWebhookAction test
Mute FlushIT tests
Add “took” timing info to response for _msearch/template API (#30961)
This code is from an Apache 2.0 licensed codebase and when we imported
it into our codebase it carried the Apache 2.0 license as well. However,
during the migration of the X-Pack codebase from the internal private
repository to the elastic/elasticsearch repository, the migration tool
mistakently changed the license on this source file from the Apache 2.0
license to the Elastic license. This commit addresses this mistake by
reapplying the Apache 2.0 license.
* es/master: (24 commits)
Add missing_bucket option in the composite agg (#29465)
Rename index_prefix to index_prefixes (#30932)
Rename methods in PersistentTasksService (#30837)
[DOCS] Fix watcher file location
Update the version checks around range bucket keys, now that the change was backported.
Use dedicated ML APIs in tests (#30941)
[DOCS] Remove reference to platinum Docker image (#30916)
Minor clean-up in InternalRange. (#30886)
stable filemode for zip distributions (#30854)
[DOCS] Adds missing TLS settings for auditing (#30822)
[test] packaging: use shell when running commands (#30852)
Fix location of AbstractHttpServerTransport (#30888)
[test] packaging test logging for suse distros
Moved keyword tokenizer to analysis-common module (#30642)
Upgrade to Lucene-7.4-snapshot-1cbadda4d3 (#30928)
Limit the scope of BouncyCastle dependency (#30358)
[DOCS] Reset edit links (#30909)
Fix IndexTemplateMetaData parsing from xContent (#30917)
Remove log traces in AzureStorageServiceImpl and fix test (#30924)
Deprecate accepting malformed requests in stored script API (#28939)
...
This commit renames methods in the PersistentTasksService, to
make obvious that the methods send requests in order to change
the state of persistent tasks.
Relates to #29608.
ML has dedicated APIs for datafeeds and jobs yet base test classes and
some tests were relying on the cluster state for this state. This commit
removes this usage in favor of using the dedicated endpoints.
Limits the scope of the runtime dependency on
BouncyCastle so that it can be eventually removed.
* Splits functionality related to reading and generating certificates
and keys in two utility classes so that reading certificates and
keys doesn't require BouncyCastle.
* Implements a class for parsing PEM Encoded key material (which also
adds support for reading PKCS8 encoded encrypted private keys).
* Removes BouncyCastle dependency for all of our test suites(except
for the tests that explicitly test certificate generation) by using
pre-generated keys/certificates/keystores.
* master:
silence InstallPluginCommandTests, see https://github.com/elastic/elasticsearch/issues/30900
Remove left-over comment
Fix double semicolon in import statement
[TEST] Fix minor random bug from #30794
Include size of snapshot in snapshot metadata #18543, bwc clean up (#30890)
Enabling testing against an external cluster (#30885)
Add public key header/footer (#30877)
SQL: Remove the last remaining server dependencies from jdbc (#30771)
Include size of snapshot in snapshot metadata (#29602)
Do not serialize basic license exp in x-pack info (#30848)
Change BWC version for VerifyRepositoryResponse (#30796)
[DOCS] Document index name limitations (#30826)
Harmonize include_defaults tests (#30700)
The shard changes api returns the minimum IndexMetadata version the leader
index needs to have. If the leader side is behind on IndexMetadata version
then follow shard task waits with processing write operations until the
mapping has been fetched from leader index and applied in follower index
in the background.
The cluster state api is used to fetch the leader mapping and put mapping api
to apply the mapping in the follower index. This works because put mapping
api accepts fields that are already defined.
Relates to #30086
Removes the last remaining server dependencies from jdbc client. In order to do that it introduces the new project sql-shared-proto that contains only XContent-serializable classes. HTTP Client and JDBC now depend only on sql-shared-proto. I had to keep the original sql-proto project since it is used as a dependency by sql-cli and security integration tests.
Relates #29856
This is a bug that was identified by the kibana team. Currently on a
get-license call we do not serialize the hard-coded expiration for basic
licenses. However, the kibana team calls the x-pack info route which
still does serialize the expiration date. This commit removes that
serialization in the rest response.
* es/master:
Move score script context from SearchScript to its own class (#30816)
Fix bad version check writing Repository nodes (#30846)
[docs] explainer for java packaging tests (#30825)
Remove Throwable usage from transport modules (#30845)
REST high-level client: add put ingest pipeline API (#30793)
Update the version checks around ip_range bucket keys, now that the change was backported.
Mute IndexMasterFailoverIT.testMasterFailoverDuringIndexingWithMappingChanges
Use geohash cell instead of just a corner in geo_bounding_box (#30698)
Limit user to single concurrent auth per realm (#30794)
[Tests] Move templated _rank_eval tests (#30679)
Security: fix dynamic mapping updates with aliases (#30787)
Ensure that ip_range aggregations always return bucket keys. (#30701)
Use remote client in TransportFieldCapsAction (#30838)
Move Watcher versioning setting to meta field (#30832)
[Docs] Explain incomplete dates in range queries (#30689)
Move persistent task registrations to core (#30755)
Decouple ClusterStateTaskListener & ClusterApplier (#30809)
Send client headers from TransportClient (#30803)
Packaging: Ensure upgrade_is_oss flag file is always deleted (#30732)
Force stable file modes for built packages (#30823)
Currently nio and netty modules use the CompletableFuture class for
managing listeners. This is unfortunate as that class accepts
Throwable. This commit adds a class CompletableContext that wraps
the CompletableFuture but does not accept Throwable. This allows the
modification of netty and nio logic to no longer handle Throwable.
This commit reworks the way our realms perform caching in order to
limit each principal to a single ongoing authentication per realm. In
other words, this means that multiple requests made by the same user
will not trigger more that one authentication attempt at a time if no
entry has been stored in the cache. If an entry is present in our
cache, there is no restriction on the number of concurrent
authentications performed for this user.
This change enables us to limit the load we place on an external system
like an LDAP server and also preserve resources such as CPU on
expensive operations such as BCrypt authentication.
Closes#30355
This commit fixes an issue with dynamic mapping updates when an index
operation is performed against an alias and when the user only has
permissions to the alias. Dynamic mapping updates resolve the concrete
index early to prevent issues so the information about the alias that
the triggering operation was being executed against is lost. When
security is enabled and a user only has privileges to the alias, this
dynamic mapping update would be rejected as it is executing against the
concrete index and not the alias. In order to handle this situation,
the security code needs to look at the concrete index and the
authorized indices of the user; if the concrete index is not authorized
the code will attempt to find an alias that the user has permissions to
update the mappings of.
Closes#30597
The .watcher-history-* template is currently using a plugin-custom index setting xpack.watcher.template.version,
which prevents this template from being installed in a mixed OSS / X-Pack cluster, ultimately
leading to the situation where an X-Pack node is constantly spamming an OSS master with (failed)
template updates. Other X-Pack templates (e.g. security-index-template or security_audit_log)
achieve the same versioning functionality by using a custom _meta field in the mapping instead.
This commit switches the .watcher-history-* template to use the _meta field instead.
Persistent tasks was moved from X-Pack to core in #28455.
However, registration of the named writables and named
X-content was left in X-Pack.
This change moves the registration of the named writables
and named X-content into core. Additionally, the persistent
task actions are no longer registered in the X-Pack client
plugin, as they are already registered in ActionModule.
This change adds a simple header to the transport client
that is present on the servers thread context that ensures
we can detect if a transport client talks to the server in a
specific request. This change also adds a header for xpack
to detect if the client has xpack installed.
* master:
[DOCS] Fixes typos in security settings
Fix GeoShapeQueryBuilder serialization after backport
[DOCS] Splits auditing.asciidoc into smaller files
Reintroduce mandatory http pipelining support (#30820)
Painless: Types Section Clean Up (#30283)
Add support for indexed shape routing in geo_shape query (#30760)
[test] java tests for archive packaging (#30734)
Revert "Make http pipelining support mandatory (#30695)" (#30813)
[DOCS] Fix more edit URLs in Stack Overview (#30704)
Use correct cluster state version for node fault detection (#30810)
Change serialization version of doc-value fields.
[DOCS] Fixes broken link for native realm
[DOCS] Clarified audit.index.client.hosts (#30797)
[TEST] Don't expect acks when isolating nodes
Add a `format` option to `docvalue_fields`. (#29639)
Fixes UpdateSettingsRequestStreamableTests mutate bug
Mustes {p0=snapshot.get_repository/10_basic/*} YAML test
Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled"
Only allow x-pack metadata if all nodes are ready (#30743)
Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled
Use original settings on full-cluster restart (#30780)
Only ack cluster state updates successfully applied on all nodes (#30672)
Expose Lucene's FeatureField. (#30618)
Fix a grammatical error in the 'search types' documentation.
Remove http pipelining from integration test case (#30788)
This commit reintroduces 31251c9 and 63a5799. These commits introduced a
memory leak and were reverted. This commit brings those commits back
and fixes the memory leak by removing unnecessary retain method calls.
This reverts commit 31251c9 introduced in #30695.
We suspect this commit is causing the OOME's reported in #30811 and we will use this PR to test this assertion.
This commit adds the ability to configure how a docvalue field should be
formatted, so that it would be possible eg. to return a date field
formatted as the number of milliseconds since Epoch.
Closes#27740
Enables a rolling restart from the OSS distribution to the x-pack based distribution by preventing
x-pack code from installing custom metadata into the cluster state until all nodes are capable of
deserializing this metadata.
When doing a node restart using the test framework, the restarted node does not only use the
settings provided to the original node, but also additional settings provided by plugin extensions,
which does not correspond to the settings that a node would have on a true restart.
This is related to #29500. We are removing the ability to disable http
pipelining. This PR removes the references to disabling pipelining in
the integration test case.
* master:
QA: Add xpack tests to rolling upgrade (#30795)
Modify state of VerifyRepositoryResponse for bwc (#30762)
Reduce CLI scripts to one-liners on Windows (#30772)
Simplify number of shards setting (#30783)
Replace Request#setHeaders with addHeader (#30588)
[TEST] remove endless wait in RestClientTests (#30776)
[Docs] Fix script-fields snippet execution (#30693)
Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 (#30778)
[DOCS] Add SAML configuration information (#30548)
[DOCS] Remove X-Pack references from SQL CLI (#30694)
Make http pipelining support mandatory (#30695)
[Docs] Fix typo in circuit breaker docs (#29659)
[Feature] Adding a char_group tokenizer (#24186)
[Docs] Fix broken cross link in documentation
Test: wait for netty threads in a JUnit ClassRule (#30763)
Increase the maximum number of filters that may be in the cache. (#30655)
[Security] Include an empty json object in an json array when FLS filters out all fields (#30709)
[TEST] Wait for CS to be fully applied in testDeleteCreateInOneBulk
Add more yaml tests for get alias API (#29513)
Ignore empty completion input (#30713)
[DOCS] fixed incorrect default
[ML] Filter undefined job groups from update calendar actions (#30757)
Fix docs failure on language analyzers (#30722)
[Docs] Fix inconsistencies in snapshot/restore doc (#30480)
Enable installing plugins from snapshots.elastic.co (#30765)
Remove fedora 26, add 28 (#30683)
Accept Gradle build scan agreement (#30645)
Remove logging from elasticsearch-nio jar (#30761)
Add Delete Repository High Level REST API (#30666)
This commit reduces the Windows CLI scripts to one-liners by moving all
of the redundant logic to an elasticsearch-cli script. This commit is
only the Windows side, a previous commit covered the Linux side.
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.
Closes#30741Closes#30577
This commit changes the wait for a few netty threads to wait for these
threads to complete after the cluster has stopped. Previously, we were
waiting for these threads before the cluster was actually stopped; the
cluster is stopped in an AfterClass method of ESIntegTestCase, while
the wait was performed in the AfterClass of a class that extended
ESIntegTestCase, which is always executed before the AfterClass of
ESIntegTestCase.
Now, the wait is contained in an ExternalResource ClassRule that
implements the waiting for the threads to terminate in the after
method. This rule is executed after the AfterClass method in
ESIntegTestCase. The same fix has also been applied in
SecuritySingleNodeTestCase.
Closes#30563
Prior to this change an json array element with no fields would be omitted from json array.
Nested inner hits source filtering relies on the fact that the json array element numbering
remains untouched and this causes AOOB exceptions in the ES side during the fetch phase
without this change.
Closes#30624
