* master:
TEST: testDocStats should always use forceMerge (#32450)
TEST: Avoid deletion in FlushIT
AwaitsFix IndexShardTests#testDocStats
Painless: Add method type to method. (#32441)
Due to the recent change in LUCENE-8263, we need to adjust the deletion
ration to between 10% to 33% to preserve the current behavior of the
test. However, we may need another refinement if soft-deletes is enabled
as the actual deletes are different because of delete tombstones.
This commit prefers to always execute forceMerge instead of adjusting
the deletion ratio so that this test can focus on testing docStats.
Closes#32449
Due to the recent change in LUCENE-8263, a merge can be triggered if the
deletion ration is higher than 33%. An in-progress merge can prevent a
synced-flush from issuing.
This commit avoids deletes by using different docIds.
Closes#32436
MethodType can be computed at compile-time rather than run-time. This removes the
method that collects MethodType at run-time from a PainlessMethod since is it no longer
necessary.
* master:
Remove reference to non-existent store type (#32418)
[TEST] Mute failing FlushIT test
Fix ordering of bootstrap checks in docs (#32417)
[TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints
[TEST] Mute failing testConvertLongHexError
bump lucene version after backport
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
[Kerberos] Avoid vagrant update on precommit (#32416)
TESTS: Move netty leak detection to paranoid level (#32354)
[DOCS] Fixes formatting of scope object in job resource
Copy missing segment attributes in getSegmentInfo (#32396)
AbstractQueryTestCase should run without type less often (#28936)
INGEST: Fix Deprecation Warning in Script Proc. (#32407)
Switch x-pack/plugin to new style Requests (#32327)
Docs: Correcting a typo in tophits (#32359)
Build: Stop double generating buildSrc pom (#32408)
TEST: Avoid triggering merges in FlushIT
Fix missing JavaDoc for @throws in several places in KerberosTicketValidator.
Switch x-pack full restart to new style Requests (#32294)
Release requests in cors handler (#32364)
Painless: Clean Up PainlessClass Variables (#32380)
Docs: Fix callouts in put license HL REST docs (#32363)
[ML] Consistent pattern for strict/lenient parser names (#32399)
Update update-settings.asciidoc (#31378)
Remove some dead code (#31993)
Introduce index store plugins (#32375)
Rank-Eval: Reduce scope of an unchecked supression
Make sure _forcemerge respects `max_num_segments`. (#32291)
TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests (#32377)
Only enforce password hashing check if FIPS enabled (#32383)
We removed the default_fs store type yet the docs still contain a
reference to them. This commit addresses that by removing this
reference, and changing a reference to this section of the docs to
instead refer to mmapfs.
In the section of the bootstrap checks docs for the maximum map count
check, we refer to max size virtual memory check and explicitly call out
the maximum size virtual memory check as being the previous
point. However, this is not correct as the previous point is currently
the max file size check. It does make sense for these two checks to be
proximate to each other in the docs so this commit reorders the checks
so that the maximum size virtual memory check indeed comes before the
maximum map count check. This makes the sense in the maximum map count
check correct.
Today it's possible to encounter an Index operation in Lucene whose
_source is disabled, and _recovery_source was pruned by the MergePolicy.
If it's the case, we create a Translog#Index without source and let the
caller validate it later. However, this approach is challenging for the
caller.
Deletes and No-Ops don't allow invoking "source()" method. The caller
has to make sure to call "source()" only on index operations. The
current implementation in CCR does not follow this and fail to replica
deletes or no-ops. Moreover, it's easier to reason if a Translog#Index
always has the source.
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
This commit avoids dependency during compile on copy keytab to
be present in the generated sources so pre-commit does not
stall for updating vagrant box.
Closes#32387
This commit changes the randomization to always create an index with a type.
It also adds a way to create a query shard context that maps to an index with
no type registered in order to explicitely test cases where there is no type.
* Using short script form normalized to a map that used 'inline' instead of 'source' so a short form processor definition like:
```
{
"script": "ctx.foo= 'bar'"
}
```
would always warn about the following deprecation:
```
#! Deprecation: Deprecated field [inline] used, expected [source]
```
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/plugin` project to use the new versions.
When we added the `java-gradle-plugin` to `buildSrc` it added a second
task to generate the pom that duplicates the publishing work that we
configure in `BuildPlugin`. Not only does it dupliciate the pom, it
creates a pom that is missing things like `name` and `description` which
are required for publishing to maven central.
This change disables the duplicate pom generation.
In testSyncedFlushSkipOutOfSyncReplicas, we reindex the extra documents
to all shards including the out-of-sync replica. However, reindexing to
that replica can trigger merges (due to the new deletes) which cause the
synced-flush failed. This test starts failing after we aggressively
trigger merges segments with a large number of deletes in LUCENE-8263.
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:qa:full-cluster-restart` project to use
the new versions.
There are two scenarios where a http request could terminate in the cors
handler. If that occurs, the requests need to be released. This commit
releases those requests.
Removes the variables name, clazz, and type as they are unnecessary. Renames
staticMembers -> staticFields, members -> fields, getters -> getterMethodHandles, and
setters -> setterMethodHandles.
Previously we had two patterns for naming of strict
and lenient parsers.
Some classes had CONFIG_PARSER and METADATA_PARSER,
and used an enum to pass the parser type to nested
parsers.
Other classes had STRICT_PARSER and LENIENT_PARSER
and used ternary operators to pass the parser type
to nested parsers.
This change makes all ML classes use the second of
the patterns described above.
Removing some dead code or supressing warnings where apropriate. Most of the
time the variable tested for null is dereferenced earlier or never used before.
Today we allow plugins to add index store implementations yet we are not
doing this in our new way of managing plugins as pull versus push. That
is, today we still allow plugins to push index store providers via an on
index module call where they can turn around and add an index
store. Aside from being inconsistent with how we manage plugins today
where we would look to pull such implementations from plugins at node
creation time, it also means that we do not know at a top-level (for
example, in the indices service) which index stores are available. This
commit addresses this by adding a dedicated plugin type for index store
plugins, removing the index module hook for adding index stores, and by
aggregating these into the top-level of the indices service.
An upcoming [Lucene change](https://issues.apache.org/jira/browse/LUCENE-7976)
will make TieredMergePolicy respect the maximum merged segment size all the
time, meaning it will possibly not respect the `max_num_segments` parameter
anymore if the shard is larger than the maximum segment size.
This change makes sure that `max_num_segments` is respected for now in order
to give us time to think about how to integrate this change, and also to delay
it until 7.0 as this might be a big-enough change for us to wait for a new
major version.
* TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests
* Release all ref counted things that weren't getting properly released
* Mannually force channel promise to be completed because mock channel doesn't do it and it prevents one `release` call in `io.netty.channel.ChannelOutboundHandlerAdapter#write` from firing
* master:
[DOCS] Fix formatting error in Slack action
Painless: Fix documentation links to use existing refs (#32335)
Painless: Decouple PainlessLookupBuilder and Whitelists (#32346)
[DOCS] Adds recommendation for xpack.security.enabled (#32345)
[TEST] Mute ConvertProcessortTests.testConvertIntHexError
[TEST] Fix failure due to exception message in java11 (#32321)
[DOCS] Fixes typo in ML aggregations page
[DOCS] Adds link from bucket_span property to common time units
[ML][DOCS] Add documentation for detector rules and filters (#32013)
Add opaque_id to index audit logging (#32260)
Add 6.5.0 version to master
fixes broken build for third-party-tests (#32353)
Implements a static function in PainlessLookupBuilder that contains all the logic related
to Whitelist. PainlessLookupBuilder is available for use in loading from methods beyond
Whitelist now.
Java 11 uses more verbose exceptions messages, causing this assertion
to fail. Changed the test to be less restrictive and only look
for the classes we care about.
Nhat Nguyen
Jack Conradson
Jason Tedor
javanna
Jim Ferenczi
Jim Ferenczi
Yogesh Gaikwad
Armin Braun
lcawl
Nik Everett
Sandeep Kanabar
Jack Conradson
Tim Brooks
Igor Motov
David Roberts
mzbeck
Christoph Büscher
Adrien Grand
Dimitris Athanasiou
Zachary Tong