* master:
Do not check for object existence when deleting repository index files (#31680)
Remove extra check for object existence in repository-gcs read object (#31661)
Support multiple system store types (#31650)
[Test] Clean up some repository-s3 tests (#31601)
[Docs] Use capital letters in section headings (#31678)
[DOCS] Add PQL language Plugin (#31237)
Merge AzureStorageService and AzureStorageServiceImpl and clean up tests (#31607)
TEST: Fix test task invocation (#31657)
Revert "[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg"
Fix RealmInteg test failures
Extend allowed characters for grok field names (#21745) (#31653)
[DOCS] Fix licensing API details (#31667)
[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg
Fix CreateSnapshotRequestTests Failure (#31630)
Configurable password hashing algorithm/cost (#31234)
[TEST] Mute failing NamingConventionsTaskIT tests
[DOCS] Replace CONFIG_DIR with ES_PATH_CONF (#31635)
Core: Require all actions have a Task (#31627)
Before deleting a repository index generation file, BlobStoreRepository
checks for the existence of the file and then deletes it. We can save
a request here by using BlobContainer.deleteBlobIgnoringIfNotExists()
which ignores error when deleting a file that does not exist.
Since there is no way with S3 to know if a non versioned file existed
before being deleted, this pull request also changes S3BlobContainer so
that it now implements deleteBlobIgnoringIfNotExists(). It will now save
one more request (blobExist?) when appropriate. The tests and fixture
have been modified to conform the S3 API that always returns a 204/NO
CONTENT HTTP response on deletions.
Support multiple system store types
When falling back to using the system keystore and - most usually -
truststore, do not assume that it will be a JKS store, but deduct
its type from {@code KeyStore#getDefaultKeyStoreType}. This allows
the use of any store type the Security Provider supports by setting
the keystore.type java security property.
This commit removes some tests in the repository-s3 plugin that
have not been executed for 2+ years but have been maintained
for nothing. Most of the tests in AbstractAwsTestCase were
obsolete or superseded by fixture based integration tests.
This pull request merges the AzureStorageService interface and
the AzureStorageServiceImpl classes into one single
AzureStorageService class. It also removes some tests in the
repository-azure plugin that have not been executed for 2+ years.
As part of the changes in #31234,the password verification logic
determines the algorithm used for hashing the password from the
format of the stored password hash itself. Thus, it is generally
possible to validate a password even if it's associated stored hash
was not created with the same algorithm than the one currently set
in the settings.
At the same time, we introduced a check for incoming client change
password requests to make sure that the request's password is hashed
with the same algorithm that is configured to be used in the node
settings.
In the spirit of randomizing the algorithms used, the
{@code SecurityClient} used in the {@code NativeRealmIntegTests} and
{@code ReservedRealmIntegTests} would send all requests dealing with
user passwords by randomly selecting a hashing algorithm each time.
This meant that some change password requests were using a different
password hashing algorithm than the one used for the node and the
request would fail.
This commit changes this behavior in the two aforementioned Integ
tests to use the same password hashing algorithm for the node and the
clients, no matter what the request is.
Resolves#31670
If the recovery and indexing are concurrently happening, it's possible
for a replica to receive the same operation twice: one from the
replication, and the other from the recovery. However, these operations
are not identical because we don't store the versionType of operations
in the Lucene index.
The TranslogWriter#assertNoSeqNumberConflict assertion has been tripped
several times in the CCR branch since we use Lucene in peer-recovery.
This commit relaxes that assertion by excluding the versionType from the
check.
Make password hashing algorithm/cost configurable for the
stored passwords of users for the realms that this applies
(native, reserved). Replaces predefined choice of bcrypt with
cost factor 10.
This also introduces PBKDF2 with configurable cost
(number of iterations) as an algorithm option for password hashing
both for storing passwords and for the user cache.
Password hash validation algorithm selection takes into
consideration the stored hash prefix and only a specific number
of algorithnm and cost factor options for brypt and pbkdf2 are
whitelisted and can be selected in the relevant setting.
The TaskManager and TaskAwareRequest could return null when registering
a task according to their javadocs, but no implementations ever actually
did that. This commit removes that wording from the javadocs and ensures
null is no longer allowed.
* master:
Docs: Remove duplicate test setup
Print output when the name checker IT fails (#31660)
Fix syntax errors in get-snapshots docs (#31656)
Docs: Fix description of percentile ranks example example (#31652)
Add MultiSearchTemplate support to High Level Rest client (#30836)
Add test for low-level client round-robin behaviour (#31616)
SQL: Refactor package names of sql-proto and sql-shared-proto projects (#31622)
Remove deprecation warnings to prepare for Gradle 5 (sourceSets.main.output.classesDirs) (#30389)
Correct integTest enable logic (#31646)
Fix missing get-snapshots docs reference #31645
Do not check for Azure container existence (#31617)
Merge AwsS3Service and InternalAwsS3Service in a S3Service class (#31580)
Upgrade gradle wrapper to 4.8 (#31525)
Only set vm.max_map_count if greater than default (#31512)
Add Get Snapshots High Level REST API (#31537)
QA: Merge query-builder-bwc to restart test (#30979)
Update reindex.asciidoc (#31626)
Docs: Skip xpack snippet tests if no xpack (#31619)
mute CreateSnapshotRequestTests
HLRest: Fix test for explain API
[TEST] Fix RemoteClusterConnectionTests
Add Create Snapshot to High-Level Rest Client (#31215)
Remove legacy MetaDataStateFormat (#31603)
Add explain API to high-level REST client (#31387)
Preserve thread context when connecting to remote cluster (#31574)
Unify headers for full text queries
Remove redundant 'minimum_should_match'
JDBC driver prepared statement set* methods (#31494)
[TEST] call yaml client close method from test suite (#31591)
The range docs had an introductory section that described how to set up
and index *and* a test setup section in `docs/build.gradle` that
duplicated that section. This is bad because these section can (and do)
drift from one another. This change removes the setup in build.gradle
and marks the introductor snippet with `// TESTSETUP` so it is used on
all the snippets.
Today we use INTERNAL version type for the history operations from
Lucene index. However, this does not cover the case in which the
original operation has the version type EXTERNAL and version number 0.
Semantically all operations from translog or Lucene should always use
EXTERNAL version type.
* Remove deprecation warnings to prepare for Gradle 5
Gradle replaced `project.sourceSets.main.output.classesDir` of type
`File` with `project.sourceSets.main.output.classesDirs` of type
`FileCollection`
(see [SourceSetOutput](https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/java/org/gradle/api/tasks/SourceSetOutput.java))
Build output is now stored on a per language folder.
There are a few places where we use that, here's these and how it's
fixed:
- Randomized Test execution
- look in all test folders ( pass the multi dir configuration to the
ant runner )
- DRY the task configuration by introducing `basedOn` for
`RandomizedTestingTask` DSL
- Extend the naming convention test to support passing in multiple
directories
- Fix the standalon test plugin, the dires were not passed trough,
checked with a debuger and the statement had no affect due to a
missing `=`.
Closes#30354
* Only check Java tests, PR feedback
- Name checker was ran for Groovy tests that don't adhere to the same
convections causing the check to fail
- implement PR feedback
* Replace `add` with `addAll`
This worked because the list is passed to `project.files` that does the
right thing.
* Revert "Only check Java tests, PR feedback"
This reverts commit 9bd9389875d8b88aadb50df57a45cd0d2b073241.
* Remove `basedOn` helper
* Bring some changes back
Previus revert accidentally reverted too much
* Fix negation
* add back public
* revert name check changes
* Revert "revert name check changes"
This reverts commit a2800c0b363168339ea65e2a79ec8256e5883e6d.
* Pass all dirs to name check
Only run on Java for build-tools, this is safe because it's a self test.
It needs more work before we could pass in the Groovy classes as well as
these inherit from `GroovyTestCase`
* remove self tests from name check
The self complicates the task setup and disable real checks on
build-tools.
With this change there are no more self tests, and the build-tools tests
adhere to the conventions.
The self test will be replaced by gradle test kit, thus the addition of
the Gradle plugin builder plugin.
* First test to run a Gradle build
* Add tests that replace the name check self test
* Clean up integ test base class
* Always run tests
* Align with test naming conventions
* Make integ. test case inherit from unit test case
The check requires this
* Remove `import static org.junit.Assert.*`
The current AzureStorageServiceImpl always checks if the Azure container
exists before reading or writing an object to the Azure container. This commit
removes this behavior, reducing the number of overhall requests executed
for all snapshots operations.
The interface and its implementation can be merged into a single class,
which is renamed to S3Service like the other S3BlobStore, S3Repository
classes.
* Move to Gradle 4.8 RC1
* Use latest version of plugin
The current does not work with Gradle 4.8 RC1
* Switch to Gradle GA
* Add and configure build compare plugin
* add work-around for https://github.com/gradle/gradle/issues/5692
* work around https://github.com/gradle/gradle/issues/5696
* Make use of Gradle build compare with reference project
* Make the manifest more compare friendly
* Clear the manifest in compare friendly mode
* Remove animalsniffer from buildscript classpath
* Fix javadoc errors
* Fix doc issues
* reference Gradle issues in comments
* Conditionally configure build compare
* Fix some more doclint issues
* fix typo in build script
* Add sanity check to make sure the test task was replaced
Relates to #31324. It seems like Gradle has an inconsistent behavior and
the taks is not always replaced.
* Include number of non conforming tasks in the exception.
* No longer replace test task, create implicit instead
Closes#31324. The issue has full context in comments.
With this change the `test` task becomes nothing more than an alias for `utest`.
Some of the stand alone tests that had a `test` task now have `integTest`, and a
few of them that used to have `integTest` to run multiple tests now only
have `check`.
This will also help separarate unit/micro tests from integration tests.
* Revert "No longer replace test task, create implicit instead"
This reverts commit f1ebaf7d93e4a0a19e751109bf620477dc35023c.
* Fix replacement of the test task
Based on information from gradle/gradle#5730 replace the task taking
into account the task providres.
Closes#31324.
* Only apply build comapare plugin if needed
* Make sure test runs before integTest
* Fix doclint aftter merge
* PR review comments
* Switch to Gradle 4.8.1 and remove workaround
* PR review comments
* Consolidate task ordering
So the issue here is that we want to avoid setting vm.max_map_count if
it is already equal to the desired value (the bootstrap check requires
262144). The reason we want to avoid this is because in some use-cases
using sysctl to set this will fail. In this case, we want to enable
users to set this value externally and then allow that to cause using
sysctl to set the value to be skipped so that cases where using sysctl
will fail to no longer fail.
Merges the `query-builder-bwc` qa project into the
`full-cluster-restart` qa project, saving a cluster starts on every
build and *many* cluster starts on `./gradlew bwcTests`.
Skips tests the require xpack if we run the doc build without xpack. So
this should work:
```
./gradlew -p docs check -Dtests.distribution=oss-zip
```
This is implemented by detecting parts of the doc that look like:
```
[testenv="basic"]
```
Relates to #30665
The test could sometimes generate an empty array of random stored fields
to fetch which would cause the test to fail. It is fine not to allow
empty here because we already randomize whether or not to generate the
stored fields param anyway.
With PR #31574 we now ensure that connections are established under the proper
thread context. There is a test in RemoteClusterConnectionTests, however,
that shuts down the service while connecting. With the above change, a new kind
of exception can occur that the test is unaware of.
Added support to the high-level rest client for the create snapshot API call. This required
several changes to toXContent which may need to be cleaned up in a later PR. Also
added several parsers for fromXContent to be able to retrieve appropriate responses
along with tests.
Establishing remote cluster connections uses a queue to coordinate multiple concurrent connect
attempts. Connect attempts can be initiated by user triggered searches as well as by system events
(e.g. when nodes disconnect). Multiple such concurrent events can lead to the connectListener of
one event to be called under the thread context of another connect attempt. This can lead to the
situation as seen in #31462 where the connect listener is executed under the system context, which
breaks when fetching the search shards from the remote cluster.
Closes#31462
Nhat Nguyen
Tanguy Leroux
Ioannis Kakavas
Fredrik Meyer
Martijn van Groningen
ritesh-kapoor
Armin Braun
Ioannis Kakavas
Lisa Cawley
Zachary Tong
Jack Conradson
Ryan Ernst
Nik Everett
Alpar Torok
Tim Brooks
Peter Evers
markharwood
Luca Cavanna
Igor Motov
Ben Abrams
DeDe Morton
Vladimir Dolzhenko
Yannick Welsch
Yu