* master:
silence InstallPluginCommandTests, see https://github.com/elastic/elasticsearch/issues/30900
Remove left-over comment
Fix double semicolon in import statement
[TEST] Fix minor random bug from #30794
Include size of snapshot in snapshot metadata #18543, bwc clean up (#30890)
Enabling testing against an external cluster (#30885)
Add public key header/footer (#30877)
SQL: Remove the last remaining server dependencies from jdbc (#30771)
Include size of snapshot in snapshot metadata (#29602)
Do not serialize basic license exp in x-pack info (#30848)
Change BWC version for VerifyRepositoryResponse (#30796)
[DOCS] Document index name limitations (#30826)
Harmonize include_defaults tests (#30700)
Include size of snapshot in snapshot metadata
Adds difference of number of files (and file sizes) between prev and current snapshot. Total number/size reflects total number/size of files in snapshot.
Closes#18543
* es/master:
Move score script context from SearchScript to its own class (#30816)
Fix bad version check writing Repository nodes (#30846)
[docs] explainer for java packaging tests (#30825)
Remove Throwable usage from transport modules (#30845)
REST high-level client: add put ingest pipeline API (#30793)
Update the version checks around ip_range bucket keys, now that the change was backported.
Mute IndexMasterFailoverIT.testMasterFailoverDuringIndexingWithMappingChanges
Use geohash cell instead of just a corner in geo_bounding_box (#30698)
Limit user to single concurrent auth per realm (#30794)
[Tests] Move templated _rank_eval tests (#30679)
Security: fix dynamic mapping updates with aliases (#30787)
Ensure that ip_range aggregations always return bucket keys. (#30701)
Use remote client in TransportFieldCapsAction (#30838)
Move Watcher versioning setting to meta field (#30832)
[Docs] Explain incomplete dates in range queries (#30689)
Move persistent task registrations to core (#30755)
Decouple ClusterStateTaskListener & ClusterApplier (#30809)
Send client headers from TransportClient (#30803)
Packaging: Ensure upgrade_is_oss flag file is always deleted (#30732)
Force stable file modes for built packages (#30823)
Currently nio and netty modules use the CompletableFuture class for
managing listeners. This is unfortunate as that class accepts
Throwable. This commit adds a class CompletableContext that wraps
the CompletableFuture but does not accept Throwable. This allows the
modification of netty and nio logic to no longer handle Throwable.
Today, the `ClusterApplier` and `MasterService` both use the
`ClusterStateTaskListener` interface to notify their callers when asynchronous
activities have completed. However, this is not wholly appropriate: none of the
callers into the `ClusterApplier` care about the `ClusterState` arguments that
they receive. This change introduces a dedicated ClusterApplyListener
interface for callers into the `ClusterApplier`, to distinguish these listeners
from the real `ClusterStateTaskListener`s that are waiting for responses from
the `MasterService`.
* master:
[DOCS] Fixes typos in security settings
Fix GeoShapeQueryBuilder serialization after backport
[DOCS] Splits auditing.asciidoc into smaller files
Reintroduce mandatory http pipelining support (#30820)
Painless: Types Section Clean Up (#30283)
Add support for indexed shape routing in geo_shape query (#30760)
[test] java tests for archive packaging (#30734)
Revert "Make http pipelining support mandatory (#30695)" (#30813)
[DOCS] Fix more edit URLs in Stack Overview (#30704)
Use correct cluster state version for node fault detection (#30810)
Change serialization version of doc-value fields.
[DOCS] Fixes broken link for native realm
[DOCS] Clarified audit.index.client.hosts (#30797)
[TEST] Don't expect acks when isolating nodes
Add a `format` option to `docvalue_fields`. (#29639)
Fixes UpdateSettingsRequestStreamableTests mutate bug
Mustes {p0=snapshot.get_repository/10_basic/*} YAML test
Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled"
Only allow x-pack metadata if all nodes are ready (#30743)
Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled
Use original settings on full-cluster restart (#30780)
Only ack cluster state updates successfully applied on all nodes (#30672)
Expose Lucene's FeatureField. (#30618)
Fix a grammatical error in the 'search types' documentation.
Remove http pipelining from integration test case (#30788)
This commit reintroduces 31251c9 and 63a5799. These commits introduced a
memory leak and were reverted. This commit brings those commits back
and fixes the memory leak by removing unnecessary retain method calls.
This reverts commit 31251c9 introduced in #30695.
We suspect this commit is causing the OOME's reported in #30811 and we will use this PR to test this assertion.
When doing a node restart using the test framework, the restarted node does not only use the
settings provided to the original node, but also additional settings provided by plugin extensions,
which does not correspond to the settings that a node would have on a true restart.
This is related to #29500. We are removing the ability to disable http
pipelining. This PR removes the references to disabling pipelining in
the integration test case.
* master:
QA: Add xpack tests to rolling upgrade (#30795)
Modify state of VerifyRepositoryResponse for bwc (#30762)
Reduce CLI scripts to one-liners on Windows (#30772)
Simplify number of shards setting (#30783)
Replace Request#setHeaders with addHeader (#30588)
[TEST] remove endless wait in RestClientTests (#30776)
[Docs] Fix script-fields snippet execution (#30693)
Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 (#30778)
[DOCS] Add SAML configuration information (#30548)
[DOCS] Remove X-Pack references from SQL CLI (#30694)
Make http pipelining support mandatory (#30695)
[Docs] Fix typo in circuit breaker docs (#29659)
[Feature] Adding a char_group tokenizer (#24186)
[Docs] Fix broken cross link in documentation
Test: wait for netty threads in a JUnit ClassRule (#30763)
Increase the maximum number of filters that may be in the cache. (#30655)
[Security] Include an empty json object in an json array when FLS filters out all fields (#30709)
[TEST] Wait for CS to be fully applied in testDeleteCreateInOneBulk
Add more yaml tests for get alias API (#29513)
Ignore empty completion input (#30713)
[DOCS] fixed incorrect default
[ML] Filter undefined job groups from update calendar actions (#30757)
Fix docs failure on language analyzers (#30722)
[Docs] Fix inconsistencies in snapshot/restore doc (#30480)
Enable installing plugins from snapshots.elastic.co (#30765)
Remove fedora 26, add 28 (#30683)
Accept Gradle build scan agreement (#30645)
Remove logging from elasticsearch-nio jar (#30761)
Add Delete Repository High Level REST API (#30666)
Adding headers rather than setting them all at once seems more
user-friendly and we already do it in a similar way for parameters
(see Request#addParameter).
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.
Closes#30741Closes#30577
This is related to #29500 and #28898. This commit removes the abilitiy
to disable http pipelining. After this commit, any elasticsearch node
will support pipelined requests from a client. Additionally, it extracts
some of the http pipelining work to the server module. This extracted
work is used to implement pipelining for the nio plugin.
This is related to #27260. The elasticsearch-nio jar is supposed to be
a library opposed to a framework. Currently it internally logs certain
exceptions. This commit modifies it to not rely on logging. Instead
exception handlers are passed by the applications that use the jar.
* master:
Scripting: Remove getDate methods from ScriptDocValues (#30690)
Upgrade to Lucene-7.4.0-snapshot-59f2b7aec2 (#30726)
[Docs] Fix single page :docs:check invocation (#30725)
Docs: Add uptasticsearch to list of clients (#30738)
[DOCS] Removes out-dated x-pack/docs/en/index.asciidoc
[DOCS] Removes redundant index.asciidoc files (#30707)
[TEST] Reduce forecast overflow to disk test memory limit (#30727)
Plugins: Remove meta plugins (#30670)
[DOCS] Moves X-Pack configurationg pages in table of contents (#30702)
TEST: Add engine log to testCorruptFileThenSnapshotAndRestore
[ML][TEST] Fix bucket count assertion in ModelPlotsIT (#30717)
[ML][TEST] Make AutodetectMemoryLimitIT less fragile (#30716)
Default copy settings to true and deprecate on the REST layer (#30598)
[Build] Add test admin when starting gradle run with trial license and
This implementation lazily (on 1st forecast request) checks for available diskspace and creates a subfolder for storing data outside of Lucene indexes, but as part of the ES data paths.
Tests: Fail if test watches could not be triggered (#30392)
[ML] add version information in case of crash of native ML process (#30674)
Make TransportClusterStateAction abide to our style (#30697)
Change required version for Get Settings transport API changes to 6.4.0 (#30706)
[DOCS] Fixes edit URLs for stack overview (#30583)
Silence sleep based watcher test
[TEST] Adjust version skips for movavg/movfn tests
[DOCS] Replace X-Pack terms with attributes
[ML] Clean left behind model state docs (#30659)
Correct typos
filters agg docs duplicated 'bucket' word removal (#30677)
top_hits doc example description update (#30676)
[Docs] Replace InetSocketTransportAddress with TransportAdress (#30673)
[TEST] Account for increase in ML C++ memory usage (#30675)
User proper write-once semantics for GCS repository (#30438)
Remove bogus file accidentally added
Add detailed assert message to IndexAuditUpgradeIT (#30669)
Adjust fast forward for token expiration test (#30668)
Improve explanation in rescore (#30629)
Deprecate `nGram` and `edgeNGram` names for ngram filters (#30209)
Watcher: Fix watch history template for dynamic slack attachments (#30172)
Fix _cluster/state to always return cluster_uuid (#30656)
[Tests] Add debug information to CorruptedFileIT
# Conflicts:
# test/framework/src/main/java/org/elasticsearch/indices/analysis/AnalysisFactoryTestCase.java
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
* es/master: (74 commits)
Preserve REST client auth despite 401 response (#30558)
[test] packaging: add windows boxes (#30402)
Make xpack modules instead of a meta plugin (#30589)
Mute ShrinkIndexIT
[ML] DeleteExpiredDataAction should use client with origin (#30646)
Reindex: Fixed typo in assertion failure message (#30619)
[DOCS] Fixes list of unconverted snippets in build.gradle
[DOCS] Reorganizes RBAC documentation
SQL: Remove dependency for server's version from JDBC driver (#30631)
Test: increase search logging for LicensingTests
Adjust serialization version in IndicesOptions
[TEST] Fix compilation
Remove version argument in RangeFieldType (#30411)
Remove unused DirectoryUtils class. (#30582)
Mitigate date histogram slowdowns with non-fixed timezones. (#30534)
Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg (#29594)
Removes AwaitsFix on IndicesOptionsTests
Template upgrades should happen in a system context (#30621)
Fix bug in BucketMetrics path traversal (#30632)
Fixes IndiceOptionsTests to serialise correctly (#30644)
...
Date histograms on non-fixed timezones such as `Europe/Paris` proved much slower
than histograms on fixed timezones in #28727. This change mitigates the issue by
using a fixed time zone instead when shard data doesn't cross a transition so
that all timestamps share the same fixed offset. This should be a common case
with daily indices.
NOTE: Rewriting the aggregation doesn't work since the timezone is then also
used on the coordinating node to create empty buckets, which might be out of the
range of data that exists on the shard.
NOTE: In order to be able to get a shard context in the tests, I reused code
from the base query test case by creating a new parent test case for both
queries and aggregations: `AbstractBuilderTestCase`.
Mitigates #28727
This pipeline aggregation gives the user the ability to script functions that "move" across a window
of data, instead of single data points. It is the scripted version of MovingAvg pipeline agg.
Through custom script contexts, we expose a number of convenience methods:
- MovingFunctions.max()
- MovingFunctions.min()
- MovingFunctions.sum()
- MovingFunctions.unweightedAvg()
- MovingFunctions.linearWeightedAvg()
- MovingFunctions.ewma()
- MovingFunctions.holt()
- MovingFunctions.holtWinters()
- MovingFunctions.stdDev()
The user can also define any arbitrary logic via their own scripting, or combine with the above methods.
This change adds a `listTasks` method to the high level java
ClusterClient which allows listing running tasks through the
task management API.
Related to #27205
A 6.x node can send a deprecation message that the default number of
shards will change from five to one in 7.0.0. In a mixed cluster,
whether or not a create index request sees five or one shard and
produces a deprecation message depends on the version of the master
node. This means that during BWC tests a test can see this deprecation
message depending on the version of the master node. In 6.x when we
introduced this deprecation message we assumed that whereever we see
this deprecation message is expected. However, in a mixed cluster test
we need a similar mechanism but it would only apply if the version of
the master node is earlier than 7.0.0. This commit takes advantage of a
recent change to expose the version of the master node to do sections of
REST tests. With this in hand, we can skip asserting on the deprecation
message if the version of the master node is before 7.0.0 and otherwise
seeing that deprecation message would be completely unexpected.
This commit is related to #28898. It adds an nio driven http server
transport. Currently it only supports basic http features. Cors,
pipeling, and read timeouts will need to be added in future PRs.
Since #29458, we use a searcher to calculate the number of documents for
a commit stats. Sadly, that approach is flawed. The searcher might no
longer point to the last commit if it's refreshed. As synced-flush
requires an exact numDocs to work correctly, we have to exclude all
soft-deleted docs.
This commit makes synced-flush stop using CommitStats but read an exact
numDocs directly from an index commit.
Relates #29458
Relates #29530
This commit exposes the master version to the REST test context. This
will be needed in a follow-up where the master version will be used to
determine whether or not a certain warning header is expected.
This configures all `qa` projects to use the distribution contained in
the `tests.distribution` system property if it is set. The goal is to
create a simple way to run tests against the default distribution which
has x-pack basic features enabled while not forcing these tests on all
contributors. You run these tests by doing something like:
```
./gradlew -p qa -Dtests.distribution=zip check
```
or
```
./gradlew -p qa -Dtests.distribution=zip bwcTest
```
x-pack basic *shouldn't* get in the way of any of these tests but
nothing is ever perfect so this we have to disable a few when running
with the zip distribution.
* master:
Default to one shard (#30539)
Unmute IndexUpgradeIT tests
Forbid expensive query parts in ranking evaluation (#30151)
Docs: Update HighLevelRestClient migration docs (#30544)
Clients: Switch to new performRequest (#30543)
[TEST] Fix typo in MovAvgIT test
Add missing dependencies on testClasses (#30527)
[TEST] Mute ML test that needs updating to following ml-cpp changes
Document woes between auto-expand-replicas and allocation filtering (#30531)
Moved tokenizers to analysis common module (#30538)
Adjust copy settings versions
Mute ShrinkIndexIT suite
SQL: SYS TABLES ordered according to *DBC specs (#30530)
Deprecate not copy settings and explicitly disallow (#30404)
[ML] Improve state persistence log message
Build: Add mavenPlugin cluster configuration method (#30541)
Re-enable FlushIT tests
Bump Gradle heap to 2 GB (#30535)
SQL: Use request flavored methods in tests (#30345)
Suppress hdfsFixture if there are spaces in the path (#30302)
Delete temporary blobs before creating index file (#30528)
Watcher: Remove TriggerEngine.getJobCount() (#30395)
[ML] Fix wire BWC for JobUpdate (#30512)
Use simpler write-once semantics for FS repository (#30435)
Derive max composite buffers from max content len
Use simpler write-once semantics for HDFS repository (#30439)
SQL: Improve correctness of SYS COLUMNS & TYPES (#30418)
Mute two tests in FlushIT with @AwaitsFix.
Fix incorrect template name in test case
Build: Remove legacy bwc files from xpack (#30485)
Mute UnicastZenPingTests#testSimplePings with @AwaitsFix.
Security: cleanup code in file stores (#30348)
Security: fix TokenMetaData equals and hashcode (#30347)
Mute two tests from SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.
Mute SharedClusterSnapshotRestoreIT#testSnapshotSucceedsAfterSnapshotFailure with @AwaitsFix.
SQL: Improve compatibility with MS query (#30516)
SQL: Fix parsing of dates with milliseconds (#30419)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.
Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
The following tokenizers were moved: classic, edge_ngram,
letter, lowercase, ngram, path_hierarchy, pattern, thai, uax_url_email and
whitespace.
Left keyword tokenizer factory in server module, because
normalizers directly depend on it.This should be addressed on a
follow up change.
Relates to #23658
There's no need for an extra `blobExists()` call when writing a blob to the HDFS service. The writeBlob implementation for the HDFS repository already uses the `CreateFlag.CREATE` option on the file creation, which ensures that the blob that's uploaded does not already exist. This saves one network roundtrip.
The TokenMetaData equals method compared byte arrays using `.equals` on
the arrays themselves, which is the equivalent of an `==` check. This
means that a seperate byte[] with the same contents would not be
considered equivalent to the existing one, even though it should be.
The method has been updated to use `Array#equals` and similarly the
hashcode method has been updated to call `Arrays#hashCode` instead of
calling hashcode on the array itself.
* master:
Upgrade to Lucene-7.4-snapshot-6705632810 (#30519)
add version compatibility from 6.4.0 after backport, see #30319 (#30390)
Security: Simplify security index listeners (#30466)
Add proper longitude validation in geo_polygon_query (#30497)
Remove Discovery.AckListener.onTimeout() (#30514)
Build: move generated-resources to build (#30366)
Reindex: Fold "with all deps" project into reindex (#30154)
Isolate REST client single host tests (#30504)
Solve Gradle deprecation warnings around shadowJar (#30483)
SAML: Process only signed data (#30420)
Remove BWC repository test (#30500)
Build: Remove xpack specific run task (#30487)
AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests
LLClient: Add setJsonEntity (#30447)
Expose CommonStatsFlags directly in IndicesStatsRequest. (#30163)
Silence IndexUpgradeIT test failures. (#30430)
Bump Gradle heap to 1792m (#30484)
[docs] add warning for read-write indices in force merge documentation (#28869)
Avoid deadlocks in cache (#30461)
Test: remove hardcoded list of unconfigured ciphers (#30367)
mute SplitIndexIT due to https://github.com/elastic/elasticsearch/issues/30416
Docs: Test examples that recreate lang analyzers (#29535)
BulkProcessor to retry based on status code (#29329)
Add GET Repository High Level REST API (#30362)
add a comment explaining the need for RetryOnReplicaException on missing mappings
Add `coordinating_only` node selector (#30313)
Stop forking groovyc (#30471)
Avoid setting connection request timeout (#30384)
Use date format in `date_range` mapping before fallback to default (#29310)
Watcher: Increase HttpClient parallel sent requests (#30130)
# Conflicts:
# x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/LocalStateCompositeXPackPlugin.java
This commit removes a test that we can not restore from 1.x and 2.x
repository files. This test is not needed, the version of Elasticsearch
that this commit targets can not even read index files from those
versions.
This commit adds an API to read translog snapshot from Lucene,
then cut-over from the existing translog to the new API in CCR.
Relates #30086
Relates #29530
We have a pile of documentation describing how to rebuild the built in
language analyzers and, previously, our documentation testing framework
made sure that the examples successfully built *an* analyzer but they
didn't assert that the analyzer built by the documentation matches the
built in anlayzer. Unsuprisingly, some of the examples aren't quite
right.
This adds a mechanism that tests that the analyzers built by the docs.
The mechanism is fairly simple and brutal but it seems to be working:
build a hundred random unicode sequences and send them through the
`_analyze` API with the rebuilt analyzer and then again through the
built in analyzer. Then make sure both APIs return the same results.
Each of these calls to `_anlayze` takes about 20ms on my laptop which
seems fine.
Previously only index and delete operations are indexed into Lucene,
therefore every segment should have both _id and _version terms as these
operations contain both terms. However, this is no longer guaranteed
after noop is also indexed into Lucene. A segment which contains only
no-ops does not have neither _id or _version because a no-op does not
contain these terms.
This change adds a dummy version to no-ops and makes _id terms optional
in PerThreadIDVersionAndSeqNoLookup.
Relates #30226
* elastic-master:
Watcher: Mark watcher as started only after loading watches (#30403)
Pass the task to broadcast actions (#29672)
Disable REST default settings testing until #29229 is back-ported
Correct wording in log message (#30336)
Do not fail snapshot when deleting a missing snapshotted file (#30332)
AwaitsFix testCreateShrinkIndexToN
DOCS: Correct mapping tags in put-template api
DOCS: Fix broken link in the put index template api
Add put index template api to high level rest client (#30400)
Relax testAckedIndexing to allow document updating
[Docs] Add snippets for POS stop tags default value
Move respect accept header on no handler to 6.3.1
Respect accept header on no handler (#30383)
[Test] Add analysis-nori plugin to the vagrant tests
[Docs] Fix bad link
[Docs] Fix end of section in the korean plugin docs
Expose the Lucene Korean analyzer module in a plugin (#30397)
Docs: remove transport_client from CCS role example (#30263)
[Rollup] Validate timezone in range queries (#30338)
Use readFully() to read bytes from CipherInputStream (#28515)
Fix docs Recently merged #29229 had a doc bug that broke the doc build. This commit fixes.
Test: remove cluster permission from CCS user (#30262)
Add Get Settings API support to java high-level rest client (#29229)
Watcher: Remove unneeded index deletion in tests
When deleting or creating a snapshot for a given shard, elasticsearch
usually starts by listing all the existing snapshotted files in the repository.
Then it computes a diff and deletes the snapshotted files that are not
needed anymore. During this deletion, an exception is thrown if the file
to be deleted does not exist anymore.
This behavior is challenging with cloud based repository implementations
like S3 where a file that has been deleted can still appear in the bucket for
few seconds/minutes (because the deletion can take some time to be fully
replicated on S3). If the deleted file appears in the listing of files, then the
following deletion will fail with a NoSuchFileException and the snapshot
will be partially created/deleted.
This pull request makes the deletion of these files a bit less strict, ie not
failing if the file we want to delete does not exist anymore. It introduces a
new BlobContainer.deleteIgnoringIfNotExists() method that can be used
at some specific places where not failing when deleting a file is
considered harmless.
Closes#28322
This commit introduces a soft-deletes retention merge policy based on
the global checkpoint. Some notes on this simple retention policy:
- This policy keeps all operations whose seq# is greater than the
persisted global checkpoint and configurable extra operations prior to
the global checkpoint. This is good enough for querying history changes.
- This policy is not watertight for peer-recovery. We send the
safe-commit in peer-recovery, thus we need to also send all operations
after the local checkpoint of that commit. This is analog to the min
translog generation for recovery.
- This policy is too simple to support rollback.
Relates #29530
* master:
Set the new lucene version for 6.4.0
[ML][TEST] Clean up jobs in ModelPlotIT
Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
Watcher: Ensure trigger service pauses execution (#30363)
[DOCS] Added coming qualifiers in changelog
[DOCS] Commented out empty sections in the changelog to fix the doc build. (#30372)
Security: reduce garbage during index resolution (#30180)
Make RepositoriesMetaData contents unmodifiable (#30361)
Change quad tree max levels to 29. Closes#21191 (#29663)
Test: use trial license in qa tests with security
[ML] Add integration test for model plots (#30359)
SQL: Fix bug caused by empty composites (#30343)
[ML] Account for gaps in data counts after job is reopened (#30294)
InternalEngineTests.testConcurrentOutOfOrderDocsOnReplica should use two documents (#30121)
Change signature of Get Repositories Response (#30333)
Tests: Use different watch ids per test in smoke test (#30331)
[Docs] Add term query with normalizer example
Adds Eclipse config for xpack licence headers (#30299)
Watcher: Make start/stop cycle more predictable and synchronous (#30118)
[test] add debug logging for packaging test
[DOCS] Removed X-Pack Breaking Changes
[DOCS] Fixes link to TLS LDAP info
Update versions for start_trial after backport (#30218)
Packaging: Set elasticsearch user to have non-existent homedir (#29007)
[DOCS] Fixes broken links to bootstrap user (#30349)
Fix NPE when CumulativeSum agg encounters null/empty bucket (#29641)
Make licensing FIPS-140 compliant (#30251)
[DOCS] Reorganizes authentication details in Stack Overview (#30280)
Network: Remove http.enabled setting (#29601)
Fix merging logic of Suggester Options (#29514)
[DOCS] Adds LDAP realm configuration details (#30214)
[DOCS] Adds native realm configuration details (#30215)
ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled (#30316)
[DOCS] Enables edit links for X-Pack pages (#30278)
Packaging: Unmark systemd service file as a config file (#29004)
SQL: Reduce number of ranges generated for comparisons (#30267)
Tests: Simplify VersionUtils released version splitting (#30322)
Cancelling a peer recovery on the source can leak a primary permit (#30318)
Added changelog entry for deb prerelease version change (#30184)
Convert server javadoc to html5 (#30279)
Create default ES_TMPDIR on Windows (#30325)
[Docs] Clarify `fuzzy_like_this` redirect (#30183)
Post backport of #29658.
Fix docs of the `_ignored` meta field.
Remove MapperService#types(). (#29617)
Remove useless version checks in REST tests. (#30165)
Add a new `_ignored` meta field. (#29658)
Move repository-azure fixture test to QA project (#30253)
# Conflicts:
# buildSrc/version.properties
# server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
Upgrade to lucene-7.4.0-snapshot-1ed95c097b
This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
Fix NPE when CumulativeSum agg encounters null/empty bucket
If the cusum agg encounters a null value, it's because the value is
missing (like the first value from a derivative agg), the path is
not valid, or the bucket in the path was empty.
Previously cusum would just explode on the null, but this changes it
so we only increment the sum if the value is non-null and finite.
This is safe because even if the cusum encounters all null or empty
buckets, the cumulative sum is still zero (like how the sum agg returns
zero even if all the docs were missing values)
I went ahead and tweaked AggregatorTestCase to allow testing pipelines,
so that I could delete the IT test and reimplement it as AggTests.
Closes#27544
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.
closes#12792
This commit refactors VersionUtils.resolveReleasedVersions to be
simpler, and in the process fixes the behavior to match that of
VersionCollection.groovy.
closes#30133
This commit adds a tombstone document into Lucene for every No-op.
With this change, Lucene index is expected to have a complete history
of operations like Translog. In fact, this guarantee is subjected to the
soft-deletes retention merge policy.
Relates #29530
Starting with the refactoring in https://github.com/elastic/elasticsearch/pull/22778 (released in 5.3) we may fail to properly replicate operation when a mapping update on master fails. If a bulk
operations needs a mapping update half way, it will send a request to the master before continuing
to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster
didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is
a problem because all operations that were processed so far are not replicated any more to the
replicas. Although these operations were never "acked" to the user (we threw an error) it cause the
local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge.
This PR does a couple of things:
1) Most importantly, treat *any* mapping update failure as a document level failure, meaning only
the relevant indexing operation will fail.
2) Removes the mapping update callbacks from `IndexShard.applyIndexOperationOnPrimary` and
similar methods for simpler execution. We don't use exceptions any more when a mapping
update was successful.
I think we need to do more work here (the fact that a single slow node can prevent those mappings
updates from being acked and thus fail operations is bad), but I want to keep this as small as I can
(it is already too big).
`javadoc` will switch from detaulting to html4 to html5 in "a future
release". We should get ahead of it so we're not surprised. Also, HTML5
is the future! Er, the present. Anyway, this follows up from #30220 to
make the Javadoc for two of the four remaining projects HTML5
compatible.
Today, when processing out of order operations, we only add it into
translog but skip adding into Lucene. Translog, therefore, has a
complete history of sequence numbers while Lucene does not.
Since we would like to have a complete history in Lucene, this change
makes sure that stale operations will be added to Lucene as soft-deleted
documents if required.
Relates #29530
* es/master: (32 commits)
TEST: Unmute testPrimaryRelocationWhileIndexing
Remove remaining tribe node references (#29574)
Never leave stale delete tombstones in version map (#29619)
Do not serialize common stats flags using ordinal (#29600)
Remove stale comment from JVM stats (#29625)
TEST: Mute testPrimaryRelocationWhileIndexing
Remove bulk fallback for write thread pool (#29609)
Fix an incorrect reference to 'zero_terms_docs' in match_phrase queries.
Update the version compatibility for zero_terms_query in match_phrase.
Account translog location to ram usage in version map
Remove extra spaces from changelog
Add support to match_phrase query for zero_terms_query. (#29598)
Fix incorrect references to 'zero_terms_docs' in query parsing error messages. (#29599)
Build: Move java home checks to pre-execution phase (#29548)
Avoid side-effect in VersionMap when assertion enabled (#29585)
[Tests] Remove accidental logger usage
Add tests for ranking evaluation with aliases (#29452)
Deprecate use of `htmlStrip` as name for HtmlStripCharFilter (#27429)
Update plan for the removal of mapping types. (#29586)
[Docs] Add rankEval method for Jva HL client
...
Today we can use the soft-deletes feature from Lucene to maintain a
history of a document. This change simply replaces hard-deletes by
soft-deletes in Engine.
Besides marking a document as deleted, we also index a tombstone
associated with that delete operation. Storing delete tombstones allows
us to have a history of sequence-based operations which can serve in
recovery or rollback.
Relates #29530
The internal test cluster can sometimes have 0 nodes. In this situation,
the http.enabled flag will never be read, and thus no deprecation
warning will be emitted. This commit guards the deprecation warning
check in this case.
* master:
Remove the index thread pool (#29556)
Remove extra copy in ScriptDocValues.Strings
Fix full cluster restart test recovery (#29545)
Fix binary doc values fetching in _search (#29567)
Mutes failing MovAvgIT tests
Fix the assertion message for an incorrect current version. (#29572)
Fix the version ID for v5.6.10. (#29570)
Painless Spec Documentation Clean Up (#29441)
Add versions 5.6.10 and 6.2.5
[TEST] test against scaled value instead of fixed epsilon in MovAvgIT
Remove `flatSettings` support from request classes (#29560)
MapperService to wrap a single DocumentMapper. (#29511)
Fix dependency checks on libs when generating Eclipse configuration. (#29550)
Add null_value support to geo_point type (#29451)
Add documentation about the include_type_name option. (#29555)
Enforce translog access via engine (#29542)
Today the translog of an engine is exposed and can be accessed directly.
While this exposure offers much flexibility, it also causes these troubles:
- Inconsistent behavior between translog method and engine method.
For example, rolling a translog generation via an engine also trims
unreferenced files, but translog's method does not.
- An engine does not get notified when critical errors happen in translog
as the access is direct.
This change isolates translog of an engine and enforces all accesses to
translog via the engine.
The index thread pool is no longer needed as its primary use-case for
single-document indexing requests has been relieved now that
single-document indexing requests are converted to bulk indexing
requests (with a single document payload).
* es/master:
Add remote cluster client (#29495)
Ensure flush happens on shard idle
Adds SpanGapQueryBuilder in the query DSL (#28636)
Control max size and count of warning headers (#28427)
Make index APIs work without types. (#29479)
Deprecate filtering on `_type`. (#29468)
Fix auto-generated ID example format (#29461)
Fix typo in max number of threads check docs (#29469)
Add primary term to translog header (#29227)
Add a helper method to get a random java.util.TimeZone (#29487)
Move TimeValue into elasticsearch-core project (#29486)
Fix NPE in InternalGeoCentroidTests#testReduceRandom (#29481)
Build: introduce keystoreFile for cluster config (#29491)
test: Index more docs, so that it is less likely the search request does not time out.
This adds 2 testcases that test if a shard goes idle
pending (uncommitted) segments are committed and unreferenced
files will be freed.
Relates to #29482
This change adds the current primary term to the header of the current
translog file. Having a term in a translog header is a prerequisite step
that allows us to trim translog operations given the max valid seq# for
that term.
This commit also updates tests to conform the primary term invariant
which guarantees that all translog operations in a translog file have
its terms at most the term stored in the translog header.
* Add a helper method to get a random java.util.TimeZone
This adds a helper method to ESTestCase that returns a randomized
`java.util.TimeZone`. This can be used when transitioning code from Joda to the
JDK's time classes.
Some features have been deprecated since `6.0` like the `_parent` field or the
ability to have multiple types per index. This allows to remove quite some
code, which in-turn will hopefully make it easier to proceed with the removal
of types.
Currently rest-based tests do not work from the IDE, as the security
manager is configured to permit certain network operations when
using the snapshot jars compiled by gradle. We have an existing
workaround that explicitly associates a codebase with the path
from which the classes are loaded (in this case, the IDE build
directory). This PR adds the rest client to this workaround list.
* Move Streams.copy into elasticsearch-core and make a multi-release jar
This moves the method `Streams.copy(InputStream in, OutputStream out)` into the
`elasticsearch-core` project (inside the `o.e.core.internal.io` package). It
also makes this class into a multi-release class where the Java 9 equivalent
uses `InputStream#transferTo`.
This is a followup from
https://github.com/elastic/elasticsearch/pull/29300#discussion_r178147495
* Move ObjectParser into the x-content lib
This moves `ObjectParser`, `AbstractObjectParser`, and
`ConstructingObjectParser` into the libs/x-content dependency. This decoupling
allows them to be used for parsing for projects that don't want to depend on the
entire Elasticsearch jar.
Relates to #28504
* Fixes query_string query equals timezone check
This change fixes a bug where two `QueryStringQueryBuilder`s were found
to be equal if they had the same timezone set even if the query string
in the builders were different
Closes#29403
* Adds mutate function to QueryStringQueryBuilderTests
* iter
* es/master: (68 commits)
Allow using distance measure in the geo context precision (#29273)
Disable failing query in QueryBuilderBWCIT.
Fixed quote_field_suffix in query_string (#29332)
Use fixture to test repository-url module (#29355)
Remove undocumented action.master.force_local setting (#29351)
Enhance error for out of bounds byte size settings (#29338)
Fix QueryAnalyzerTests.
Fix HasChildQueryBuilderTests to not use the `classic` similarity.
[Docs] Correct javadoc of GetIndexRequest (#29364)
Make TransportRankEvalAction members final
Add awaits fix for a query analyzer test
Check presence of multi-types before validating new mapping (#29316)
Add awaits fix for HasChildQueryBuilderTests
Remove silent batch mode from install plugin (#29359)
Align cat thread pool info to thread pool config (#29195)
Track Lucene operations in engine explicitly (#29357)
Build: Fix Java9 MR build (#29312)
Reindex: Fix error in delete-by-query rest spec (#29318)
Improve similarity integration. (#29187)
Fix some query extraction bugs. (#29283)
...
This improves the way similarities are plugged in in order to:
- reject the classic similarity on 7.x indices and emit a deprecation
warning otherwise
- reject unkwown parameters on 7.x indices and emit a deprecation
warning otherwise
Even though this breaks the plugin API, I'd like to backport to 7.x so
that users can get deprecation warnings when they are doing something
that will become unsupported in the future.
Closes#23208Closes#29035
* Begin moving XContent to a separate lib/artifact
This commit moves a large portion of the XContent code from the `server` project
to the `libs/xcontent` project. For the pieces that have been moved, some
helpers have been duplicated to allow them to be decoupled from ES helper
classes. In addition, `Booleans` and `CheckedFunction` have been moved to the
`elasticsearch-core` project.
This decoupling is a move so that we can eventually make things like the
high-level REST client not rely on the entire ES jar, only the parts it needs.
There are some pieces that are still not decoupled, in particular some of the
XContent tests still remain in the server project, this is because they test a
large portion of the pluggable xcontent pieces through
`XContentElasticsearchException`. They may be decoupled in future work.
Additionally, there may be more piecese that we want to move to the xcontent lib
in the future that are not part of this PR, this is a starting point.
Relates to #28504
Removes a set of assertions in the test framework that verified that
Streamable objects could be serialized and deserialized across different
versions. When this was discussed the consensus was that this approach
has not caught many bugs in a long time and that serialization testing of
objects was best left to their respective unit and integration tests.
This commit also removes a transport interceptor that was used in
ESIntegTestCase tests to make these assertions about objects coming in
or off the wire.
Today we have a few problems with how we handle bad requests:
- handling requests with bad encoding
- handling requests with invalid value for filter_path/pretty/human
- handling requests with a garbage Content-Type header
There are two problems:
- in every case, we give an empty response to the client
- in most cases, we leak the byte buffer backing the request!
These problems are caused by a broader problem: poor handling preparing
the request for handling, or the channel to write to when the response
is ready. This commit addresses these issues by taking a unified
approach to all of them that ensures that:
- we respond to the client with the exception that blew us up
- we do not leak the byte buffer backing the request
We historically removed reading from the transaction log to get consistent
results from _GET calls. There was also the motivation that the read-modify-update
principle we apply should not be hidden from the user. We still agree on the fact
that we should not hide these aspects but the impact on updates is quite significant
especially if the same documents is updated before it's written to disk and made serachable.
This change adds back the ability to read from the transaction log but only for update calls.
Calls to the _GET API will always do a refresh if necessary to return consistent results ie.
if stored fields or DocValues Fields are requested.
Closes#26802
* es/master: (22 commits)
Fix building Javadoc JARs on JDK for client JARs (#29274)
Require JDK 10 to build Elasticsearch (#29174)
Decouple NamedXContentRegistry from ElasticsearchException (#29253)
Docs: Update generating test coverage reports (#29255)
[TEST] Fix issue with HttpInfo passed invalid parameter
Remove all dependencies from XContentBuilder (#29225)
Fix sporadic failure in CompositeValuesCollectorQueueTests
Propagate ignore_unmapped to inner_hits (#29261)
TEST: Increase timeout for testPrimaryReplicaResyncFailed
REST client: hosts marked dead for the first time should not be immediately retried (#29230)
TEST: Use different translog dir for a new engine
Make SearchStats implement Writeable (#29258)
[Docs] Spelling and grammar changes to reindex.asciidoc (#29232)
Do not optimize append-only if seen normal op with higher seqno (#28787)
[test] packaging: gradle tasks for groovy tests (#29046)
Prune only gc deletes below local checkpoint (#28790)
remove testUnassignedShardAndEmptyNodesInRoutingTable
#28745: remove extra option in the composite rest tests
Fold EngineDiskUtils into Store, for better lock semantics (#29156)
Add file permissions checks to precommit task
...
Once a document is deleted and Lucene is refreshed, we will not be able
to look up the `version/seq#` associated with that delete in Lucene. As
conflicting operations can still be indexed, we need another mechanism
to remember these deletes. Therefore deletes should still be stored in
the Version Map, even after Lucene is refreshed. Obviously, we can't
remember all deletes forever so a trimming mechanism is needed.
Currently, we remember deletes for at least 1 minute (the default GC
deletes cycle) and clean them periodically. This is, at the moment, the
best we can do on the primary for user facing APIs but this arbitrary
time limit is problematic for replicas. Furthermore, we can't rely on
the primary and replicas doing the trimming in a synchronized manner,
and failing to do so results in the replica and primary making different
decisions.
The following scenario can cause inconsistency between
primary and replica.
1. Primary index doc (index, id=1, v2)
2. Network packet issue causes index operation to back off and wait
3. Primary deletes doc (delete, id=1, v3)
4. Replica processes delete (delete, id=1, v3)
5. 1+ minute passes (GC deletes runs replica)
6. Indexing op is finally sent to the replica which no processes it
because it forgot about the delete.
We can reply on sequence-numbers to prevent this issue. If we prune only
deletes whose seqno at most the local checkpoint, a replica will
correctly remember what it needs. The correctness is explained as
follows:
Suppose o1 and o2 are two operations on the same document with seq#(o1)
< seq#(o2), and o2 arrives before o1 on the replica. o2 is processed
normally since it arrives first; when o1 arrives it should be discarded:
1. If seq#(o1) <= LCP, then it will be not be added to Lucene, as it was
already previously added.
2. If seq#(o1) > LCP, then it depends on the nature of o2:
- If o2 is a delete then its seq# is recorded in the VersionMap,
since seq#(o2) > seq#(o1) > LCP, so a lookup can find it and
determine that o1 is stale.
- If o2 is an indexing then its seq# is either in Lucene (if
refreshed) or the VersionMap (if not refreshed yet), so a
real-time lookup can find it and determine that o1 is stale.
In this PR, we prefer to deploy a single trimming strategy, which
satisfies both requirements, on primary and replicas because:
- It's simpler - no need to distinguish if an engine is running at
primary mode or replica mode or being promoted.
- If a replica subsequently is promoted, user experience is fully
maintained as that replica remembers deletes for the last GC cycle.
However, the version map may consume less memory if we deploy two
different trimming strategies for primary and replicas.
#28245 has introduced the utility class`EngineDiskUtils` with a set of methods to prepare/change
translog and lucene commit points. That util class bundled everything that's needed to create and
empty shard, bootstrap a shard from a lucene index that was just restored etc.
In order to safely do these manipulations, the util methods acquired the IndexWriter's lock. That
would sometime fail due to concurrent shard store fetching or other short activities that require the
files not to be changed while they read from them.
Since there is no way to wait on the index writer lock, the `Store` class has other locks to make
sure that once we try to acquire the IW lock, it will succeed. To side step this waiting problem, this
PR folds `EngineDiskUtils` into `Store`. Sadly this comes with a price - the store class doesn't and
shouldn't know about the translog. As such the logic is slightly less tight and callers have to do the
translog manipulations on their own.
This change refactors the composite aggregation to add an execution mode that visits documents in the order of the values
present in the leading source of the composite definition. This mode does not need to visit all documents since it can early terminate
the collection when the leading source value is greater than the lowest value in the queue.
Instead of collecting the documents in the order of their doc_id, this mode uses the inverted lists (or the bkd tree for numerics) to collect documents
in the order of the values present in the leading source.
For instance the following aggregation:
```
"composite" : {
"sources" : [
{ "value1": { "terms" : { "field": "timestamp", "order": "asc" } } }
],
"size": 10
}
```
... can use the field `timestamp` to collect the documents with the 10 lowest values for the field instead of visiting all documents.
For composite aggregation with more than one source the execution can early terminate as soon as one of the 10 lowest values produces enough
composite buckets. For instance if visiting the first two lowest timestamp created 10 composite buckets we can early terminate the collection since it
is guaranteed that the third lowest timestamp cannot create a composite key that compares lower than the one already visited.
This mode can execute iff:
* The leading source in the composite definition uses an indexed field of type `date` (works also with `date_histogram` source), `integer`, `long` or `keyword`.
* The query is a match_all query or a range query over the field that is used as the leading source in the composite definition.
* The sort order of the leading source is the natural order (ascending since postings and numerics are sorted in ascending order only).
If these conditions are not met this aggregation visits each document like any other agg.
* es/master: (50 commits)
Reject updates to the `_default_` mapping. (#29165)
Improve similarity docs. (#29089)
[Docs] Update api.asciidoc (#29166)
Docs: Add note about missing mapping for doc values field (#29036)
Fix BWC issue for PreSyncedFlushResponse
Remove BytesArray and BytesReference usage from XContentFactory (#29151)
Add pluggable XContentBuilder writers and human readable writers (#29120)
Add unreleased version 6.2.4 (#29171)
Add unreleased version 6.1.5 (#29168)
Add a note about using the `retry_failed` flag before accepting data loss (#29160)
Fix typo in percolate-query.asciidoc (#29155)
Require HTTP::Tiny 0.070 for release notes script
Set Java 9 checkstyle to depend on checkstyle conf (#28383)
REST high-level client: add clear cache API (#28866)
Docs: Add example of resetting index setting (#29048)
Plugins: Fix module name conflict check for meta plugins (#29146)
Build: Fix meta plugin bundled plugin names (#29147)
Build: Simplify rest spec hack configuration (#29149)
Build: Fix meta modules to not install as plugin in tests (#29150)
Fix javadoc warning in Strings for missing parameter description
...
* Remove BytesArray and BytesReference usage from XContentFactory
This removes the usage of `BytesArray` and `BytesReference` from
`XContentFactory`. Instead, a regular `byte[]` should be passed. To assist with
this a helper has been added to `XContentHelper` that will preserve the offset
and length from the underlying BytesReference.
This is part of ongoing work to separate the XContent parts from ES so they can
be factored into their own jar.
Relates to #28504
`$_path` is used by documentation tests to ignore a value from a
response, for example:
```
[source,js]
----
{
"count": 1,
"datafeeds": [
{
"datafeed_id": "datafeed-total-requests",
"state": "started",
"node": {
...
"attributes": {
"ml.machine_memory": "17179869184",
"ml.max_open_jobs": "20",
"ml.enabled": "true"
}
},
"assignment_explanation": ""
}
]
}
----
// TESTRESPONSE[s/"17179869184"/$body.$_path/]
```
That example shows `17179869184` in the compiled docs but when it runs
the tests generated by that doc it ignores `17179869184` and asserts
instead that there is a value in that field. This is required because we
can't predict things like "how many milliseconds will this take?" and
"how much memory will this take?".
Before this change it was impossible to use `$_path` when any component
of the path contained a `.`. This fixes the `$_path` evaluator to
properly escape `.`.
Closes#28770
* es/master: (97 commits)
Clarify requirements of strict date formats. (#29090)
Clarify that dates are always rendered as strings. (#29093)
Compilation fix for #29067
[Docs] Fix link to Grok patterns (#29088)
Store offsets in index prefix fields when stored in the parent field (#29067)
Fix starting on Windows from another drive (#29086)
Use removeTask instead of finishTask in PersistentTasksClusterService (#29055)
Added minimal docs for reindex api in java-api docs
Allow overriding JVM options in Windows service (#29044)
Clarify how to set compiler and runtime JDKs (#29101)
CLI: Close subcommands in MultiCommand (#28954)
TEST: write ops should execute under shard permit (#28966)
[DOCS] Add X-Pack upgrade details (#29038)
Revert "Improve error message for installing plugin (#28298)"
Docs: HighLevelRestClient#exists (#29073)
Validate regular expressions in dynamic templates. (#29013)
[Tests] Fix GetResultTests and DocumentFieldTests failures (#29083)
Reenable LiveVersionMapTests.testRamBytesUsed on Java 9. (#29063)
Mute failing GetResultTests and DocumentFieldTests
Improve error message for installing plugin (#28298)
...
Changes made in #28972 seems to have changed some assumptions about how
SMILE and CBOR write byte[] values and how this is tested. This changes
the generation of the randomized DocumentField values back to BytesArray
while expecting the JSON and YAML deserialisation to produce Base64
encoded strings and SMILE and CBOR to parse back BytesArray instances.
Closes#29080
Currently we have a fairly complicated logic in the engine constructor logic to deal with all the
various ways we want to mutate the lucene index and translog we're opening.
We can:
1) Create an empty index
2) Use the lucene but create a new translog
3) Use both
4) Force a new history uuid in all cases.
This leads complicated code flows which makes it harder and harder to make sure we cover all the
corner cases. This PR tries to take another approach. Constructing an InternalEngine always opens
things as they are and all needed modifications are done by static methods directly on the
directory, one at a time.
* Decouple XContentBuilder from BytesReference
This commit removes all mentions of `BytesReference` from `XContentBuilder`.
This is needed so that we can completely decouple the XContent code and move it
into its own dependency.
While this change appears large, it is due to two main changes, moving
`.bytes()` and `.string()` out of XContentBuilder itself into static methods
`BytesReference.bytes` and `Strings.toString` respectively. The rest of the
change is code reacting to these changes (the majority of it in tests).
Relates to #28504
As we have factored Elasticsearch into smaller libraries, we have ended
up in a situation that some of the dependencies of Elasticsearch are not
available to code that depends on these smaller libraries but not server
Elasticsearch. This is a good thing, this was one of the goals of
separating Elasticsearch into smaller libraries, to shed some of the
dependencies from other components of the system. However, this now
means that simple utility methods from Lucene that we rely on are no
longer available everywhere. This commit copies IOUtils (with some small
formatting changes for our codebase) into the fold so that other
components of the system can rely on these methods where they no longer
depend on Lucene.
I have long wanted an actual test that dying with dignity works. It is
tricky because if dying with dignity works, it means the test JVM dies
which is usually an abnormal condition. And anyway, how does one force a
fatal error to be thrown. I was motivated to investigate this again by
the fact that I missed a backport to one branch leading to an issue
where Elasticsearch would not successfully die with dignity. And now we
have a solution: we install a plugin that throws an out of memory error
when it receives a request. We hack the standalone test infrastructure
to prevent this from failing the test. To do this, we bypass the
security manager and remove the PID file for the node; this tricks the
test infrastructure into thinking that it does not need to stop the
node. We also bypass seccomp so that we can fork jps to make sure that
Elasticsearch really died. And to be extra paranoid, we parse the logs
of the dead Elasticsearch process to make sure it died with
dignity. Never forget.
* es/master: (48 commits)
Update bucket-sort-aggregation.asciidoc (#28937)
[Docs] REST high-level client: Fix code for most basic search request (#28916)
Improved percolator's random candidate query duel test and fixed bugs that were exposed by this:
Revert "Rescore collapsed documents (#28521)"
Build: Fix test logger NPE when no tests are run (#28929)
[TEST] AwaitsFix QueryRescorerIT.testRescoreAfterCollapse
Decouple XContentType from StreamInput/Output (#28927)
Remove BytesRef usage from XContentParser and its subclasses (#28792)
[DOCS] Correct typo in configuration (#28903)
Fix incorrect datemath example (#28904)
Add a usage example of the JLH score (#28905)
Wrap stream passed to createParser in try-with-resources (#28897)
Rescore collapsed documents (#28521)
Fix (simple)_query_string to ignore removed terms (#28871)
[Docs] Fix typo in composite aggregation (#28891)
Try if tombstone is eligable for pruning before locking on it's key (#28767)
Limit analyzed text for highlighting (improvements) (#28808)
Missing `timeout` parameter from the REST API spec JSON files (#28328)
Clarifies how query_string splits textual part (#28798)
Update outdated java version reference (#28870)
...
Today we have two test base classes that have a lot in common when it comes to testing wire and xcontent serialization: `AbstractSerializingTestCase` and `AbstractXContentStreamableTestCase`. There are subtle differences though between the two, in the way they work, what can be overridden and features that they support (e.g. insertion of random fields).
This commit introduces a new base class called `AbstractWireTestCase` which holds all of the serialization test code in common between `Streamable` and `Writeable`. It has two minimal subclasses called `AbstractWireSerializingTestCase` and `AbstractStreamableTestCase` which are specialized for `Writeable` and `Streamable`.
This commit also introduces a new test class called `AbstractXContentTestCase` for all of the xContent testing, which holds a testFromXContent method for parsing and rendering to xContent. This one can be delegated to from the existing `AbstractStreamableXContentTestCase` and `AbstractSerializingTestCase` so that we avoid code duplicate as much as possible and all these base classes offer the same functionalities in the same way. Having this last base class decoupled from the serialization testing may also help with the REST high-level client testing, as there are some classes where it's hard to implement equals/hashcode and this makes it possible to override `assertEqualInstances` for custom equality comparisons (also this base class doesn't require implementing equals/hashcode as it doesn't test such methods.
This commit is related to #27260. Currently there is a weird
relationship between channel contexts and nio channels. The selectors
use the context for read and writing. But the selector operates directly
on the nio channel for registering, closing, and connecting.
This commit works on improving this relationship. The selector operates
directly on the context which wraps the low level java.nio.channels. The
NioChannel class is simply an API that is used to interact with the
channel (sending messages from outside the selector event loop,
scheduling a close, adding listeners, etc). The context is only used
internally by the channel to implement these apis and by the selector to
perform these operations.
* es/master: (143 commits)
Revert "Disable BWC tests for build issues"
Remove AcknowledgedRestListener in favour of RestToXContentListener (#28724)
Build: Consolidate archives and packages configuration (#28760)
Skip some plugins service tests on Windows
Migrate some *ResponseTests to AbstractStreamableXContentTestCase (#28749)
Disable BWC tests for build issues
Ensure that azure stream has socket privileges (#28751)
[DOCS] Fixed broken link.
Pass InputStream when creating XContent parser (#28754)
[DOCS] Changed to use transient setting to reenabled allocation. Closes#27677
Delay path expansion on Windows
[TEST] replace randomAsciiAlphanumOfLengthBetween with randomAsciiLettersOfLengthBetween
[Tests] Extract the testing logic for Google Cloud Storage (#28576)
[Docs] Update links to java9 docs (#28750)
version set in ingest pipeline (#27573)
Revert "Add startup logging for standalone tests"
Tests: don't wait for completion while trying to get completed task
Add 5.6.9 snapshot version
[Docs] Java high-level REST client : clean up (#28703)
Updated distribution outputs in contributing docs
...
This allows us to save a bit of code, but also adds more coverage as it tests serialization which was missing in some of the existing tests. Also it requires implementing equals/hashcode and we get the corresponding tests for them for free from the base test class.
* Pass InputStream when creating XContent parser
Rather than passing the raw `BytesReference` in when creating the xcontent
parser, this passes the StreamInput (which is an InputStream), this allows us to
decouple XContent from BytesReference.
This also removes the use of `commons.Booleans` so it doesn't require more
external commons classes.
Related to #28504
* Undo boolean removal
* Enhance deprecation javadoc
Add support version and version_type in ingest pipelines
Add support for setting document version and version type in set
processor of an ingest pipeline.
* Remove log4j dependency from elasticsearch-core
This removes the log4j dependency from our elasticsearch-core project. It was
originally necessary only for our jar classpath checking. It is now replaced by
a `Consumer<String>` so that the es-core dependency doesn't have external
dependencies.
The parts of #28191 which were moved in conjunction (like `ESLoggerFactory` and
`Loggers`) have been moved back where appropriate, since they are not required
in the core jar.
This is tangentially related to #28504
* Add javadocs for `output` parameter
* Change @code to @link
Today we have several levels of indirection to acquire an Engine.Searcher.
We first acquire a the reference manager for the scope then acquire an
IndexSearcher and then create a searcher for the engine based on that.
This change simplifies the creation into a single method call instead of
3 different ones.
Previously we introduced a new parameter to `acquireIndexCommit` to
allow acquire either a safe commit or a last commit. However with the
new parameters, callers can provide a nonsense combination - flush first
but acquire the safe commit. This commit separates acquireIndexCommit
method into two different methods to avoid that problem. Moreover, this
change should also improve the readability.
Relates #28038
The REST high-level client supports now encoding of path parts, so that for instance documents with valid ids, but containing characters that need to be encoded as part of urls (`#` etc.), are properly supported. We also make sure that each path part can contain `/` by encoding them properly too.
Closes#28625
Currently the Translog constructor is capable both of opening an existing translog and creating a
new one (deleting existing files). This PR separates these two into separate code paths. The
constructors opens files and a dedicated static methods creates an empty translog.
* Move more XContent.createParser calls to non-deprecated version
Part 2
This moves more of the callers to pass in the DeprecationHandler.
Relates to #28504
* Use parser's deprecation handler where appropriate
* Use logging handler in test that uses deprecated field on purpose
* Move more XContent.createParser calls to non-deprecated version
This moves more of the callers to pass in the DeprecationHandler.
Relates to #28504
* Use parser's deprecation handler where available
Version Utils did not previously have logic that removed the last majors
minor snapshot if there was a next bugfix and maintenance bugfix
release. This adds the logic and fixes some broken assumptions in tests
as well.
relates #28505
The build.snapshot was mistakenly passed in to every snapshot version,
so when release tests were run, these versions were mistaken as released
entities and could not be found in maven, because they do not
exist. This fix removes that bug in logic, and always makes them proper
snapshots. This has a benefit of cleaning up the VersionUtilsTests
because they no longer rely on different sets of versions to check
against, which was also a bug.
Currently if a yaml test has a teardown and a test is failing then
a stash dump of a request in the teardown is logged instead of
a stash dump of a request in the test itself.
By handling the logging of stash dumps separately for setup, tests and
teardown yaml sections we shouldn't miss the stash dump of request/response
that is actually causing the yaml test to fail.
The is a follow up to #28567 changing the method used to capture stack traces, as requested
during the review. Instead of creating a throwable, we explicitly capture the stack trace of the
current thread. This should Make Jason Happy Again ™️ .
Generalizing BWC building so that there is less code to modify for a release. This ensures we do not
need to think about what major or minor version is in the gradle code. It follows the general rules of the
elastic release structure. For more information on the rules, see the VersionCollection's javadoc.
This also removes the additional bwc snapshots that will never be released, such as 6.0.2, which were
being built and tested against every time we ran bwc tests.
Additionally, it creates 4 new projects that correspond to the different types of snapshots that may exist
for a given version. Its possible to now run those individual tasks to work out bwc logic whereas
previously it was impossible and the entire suite of bwc tests had to be run to work out any logic
changes in the build tools' bwc project. Please note that if the project does not make sense for the
version that is current, that an error will be thrown from that individual project if an attempt is made to
run it.
This should allow for automating the version bumps as well, since it removes all the hardcoded version
logic from the configs.
Today we acquire a permit from the shard to coordinate between indexing operations, recoveries and other state transitions. When we leak an permit it's practically impossible to find who the culprit is. This PR add stack traces capturing for each permit so we can identify which part of the code is responsible for acquiring the unreleased permit. This code is only active when assertions are active.
The output is something like:
```
java.lang.AssertionError: shard [test][1] on node [node_s0] has pending operations:
--> java.lang.RuntimeException: something helpful 2
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:223)
at org.elasticsearch.index.shard.IndexShard.<init>(IndexShard.java:322)
at org.elasticsearch.index.IndexService.createShard(IndexService.java:382)
at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:514)
at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:143)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:552)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:529)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:231)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:498)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:495)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:482)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
--> java.lang.RuntimeException: something helpful
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:223)
at org.elasticsearch.index.shard.IndexShard.<init>(IndexShard.java:311)
at org.elasticsearch.index.IndexService.createShard(IndexService.java:382)
at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:514)
at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:143)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:552)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:529)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:231)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:498)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:495)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:482)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
```
This commit modifies the transport stats with exception test to remove
the requirement that we calculate the published address size when
comparing bytes received. This is tricky and is currently broken as we
also place the address string in the transport exception, however we do
not adjust the bytes for that.
The solution in this commit is to just serialize the transport exception
in the test and use that for the calculation.
* es/master:
Added more parameter to PersistentTaskPlugin#getPersistentTasksExecutor(...)
[Tests] Relax assertion in SuggestStatsIT (#28544)
Make internal Rounding fields final (#28532)
Fix the ability to remove old plugin
[TEST] Expand failure message for wildfly integration tests
Add 6.2.1 version constant
Remove feature parsing for GetIndicesAction (#28535)
No refresh on shard activation needed (#28013)
Improve failure message when restoring an index that already exists in the cluster (#28498)
Use right skip versions.
[Docs] Fix incomplete URLs (#28528)
Use non deprecated xcontenthelper (#28503)
Painless: Fixes a null pointer exception in certain cases of for loop usage (#28506)
* Move to non-deprecated XContentHelper.createParser(...)
This moves away from one of the now-deprecated XContentHelper.createParser
methods in favor of specifying the deprecation logger at parser creation time.
Relates to #28449
Note that this doesn't move all the `createParser` calls because some of them
use the already-deprecated method that doesn't specify the XContentType.
* Remove the deprecated (and now non-needed) createParser method
This commit switches all the modules and server test code to use the
non-deprecated `ParseField.match` method, passing in the parser's deprecation
handler or the logging deprecation handler when a parser is not available (like
in tests).
Relates to #28449
The primary currently replicates writes to all other shard copies as soon as they're added to the routing table. Initially those shards are not even ready yet to receive these replication requests, for example when undergoing a file-based peer recovery. Based on the specific stage that the shard copies are in, they will throw different kinds of exceptions when they receive the replication requests. The primary then ignores responses from shards that match certain exception types. With this mechanism it's not possible for a primary to distinguish between a situation where a replication target shard is not allocated and ready yet to receive requests and a situation where the shard was successfully allocated and active but subsequently failed.
This commit changes replication so that only initializing shards that have successfully opened their engine are used as replication targets. This removes the need to replicate requests to initializing shards that are not even ready yet to receive those requests. This saves on network bandwidth and enables features that rely on the distinction between a "not-yet-ready" shard and a failed shard.
This change adds a shallow copy method for aggregation builders. This method returns a copy of the builder replacing the factoriesBuilder and metaDada
This method is used when the builder is rewritten (AggregationBuilder#rewrite) in order to make sure that we create a new instance of the parent builder when sub aggregations are rewritten.
Relates #27782
Adds allow_partial_search_results flag to search requests with default setting = true.
When false, will error if search either timeouts, has partial errors or has missing shards rather
than returning partial search results. A cluster-level setting provides a default for search requests with no flag.
Closes#27435
This change makes sure that this function does not create field names that end with a '.', more precisely it only allows
alpha-numeric characters to compose the leaf field name.
Closes#27373
The MockUncasedHostProvider accesses nodes that are not fully built yet, where TransportService.getNode() returns null, which means that the null entries end up in the list of seedNodes that UnicastZenPing then uses.
Cluster settings shouldn't leak into the next test.
I played with failing the test if it left over any settings but that
felt like it added more ceremony then it was worth. The advantage is
that any test that intentionally wants to leave settings in place after
the test would fail and require looking at but, so far as I can tell, we
don't have any such tests.
Currently meta plugins will ask for confirmation of security policy
exceptions for each bundled plugin. This commit collects the necessary
permissions of each bundled plugin, and asks for confirmation of all of
them at the same time.
* es/master:
[Docs] Fix explanation for `from` and `size` example (#28320)
Adapt bwc version after backport #28358
Always return the after_key in composite aggregation response (#28358)
Adds test name to MockPageCacheRecycler exception (#28359)
Adds a note in the `terms` aggregation docs regarding pagination (#28360)
[Test] Fix DiscoveryNodesTests.testDeltas() (#28361)
Update packaging tests to work with meta plugins (#28336)
Remove Painless Type from MethodWriter in favor of Java Class. (#28346)
[Doc] Fixs typo in reverse-nested-aggregation.asciidoc (#28348)
Reindex: Shore up rethrottle test
Only assert single commit iff index created on 6.2
isHeldByCurrentThread should return primitive bool
[Docs] Clarify `html` encoder in highlighting.asciidoc (#27766)
Fix GeoDistance query example (#28355)
Settings: Introduce settings updater for a list of settings (#28338)
Adapt bwc version after backport #28310
This change adds the test name to the exceptions thrown by the MockPageCacheRecycler and MockBigArrays. Also, if there is more than one page/array which are not released it will add the first one as the cause of the thrown exception and the others as suppressed exceptions.
Relates to #21315
This introduces a settings updater that allows to specify a list of
settings. Whenever one of those settings changes, the whole block of
settings is passed to the consumer.
This also fixes an issue with affix settings, when used in combination
with group settings, which could result in no found settings when used
to get a setting for a namespace.
Lastly logging has been slightly changed, so that filtered settings now
only log the setting key.
Another bug has been fixed for the mock log appender, which did not
work, when checking for the exact message.
Closes#28047
* es/master:
Remove redundant argument for buildConfiguration of s3 plugin (#28281)
Completely remove Painless Type from AnalyzerCaster in favor of Java Class. (#28329)
Fix spelling error
Reindex: Wait for deletion in test
Reindex: log more on rare test failure
Ensure we protect Collections obtained from scripts from self-referencing (#28335)
[Docs] Fix asciidoc style in composite agg docs
Adds the ability to specify a format on composite date_histogram source (#28310)
Provide a better error message for the case when all shards failed (#28333)
[Test] Re-Add integer_range and date_range field types for query builder tests (#28171)
Added Put Mapping API to high-level Rest client (#27869)
Revert change that does not return all indices if a specific alias is requested via get alias api. (#28294)
Painless: Replace Painless Type with Java Class during Casts (#27847)
Notify affixMap settings when any under the registered prefix matches (#28317)
The tests for those field types were removed in #26549 because the range mapper
was moved to a module, but later this mapper was moved back to core in #27854.
This change adds back those two field types like before to the general setup in
AbstractQueryTestCase and adds some specifics to the RangeQueryBuilder and
TermsQueryBuilder tests. Also adding back an integration test in SearchQueryIT that
has been removed before but that can be kept with the mapper back in core now.
Relates to #28147
* master:
Trim down usages of `ShardOperationFailedException` interface (#28312)
Do not return all indices if a specific alias is requested via get aliases api.
[Test] Lower bwc version for rank-eval rest tests
CountedBitSet doesn't need to extend BitSet. (#28239)
Calculate sum in Kahan summation algorithm in aggregations (#27807) (#27848)
Remove the `update_all_types` option. (#28288)
Add information when master node left to DiscoveryNodes' shortSummary() (#28197)
Provide explanation of dangling indices, fixes#26008 (#26999)
In many cases we use the `ShardOperationFailedException` interface to abstract an exception that can only be of one type, namely `DefaultShardOperationException`. There is no need to use the interface in such cases, the concrete type should be used instead. That has the additional advantage of simplifying parsing such exceptions back from rest responses for the high-level REST client
* es/master: (38 commits)
Build: Add pom generation to meta plugins (#28321)
Add 6.3 version constant to master
Minor improvements to translog docs (#28237)
[Docs] Remove typo in painless-getting-started.asciidoc
Build: Fix meta plugin usage in integ test clusters (#28307)
Painless: Add spi jar that will be published for extending whitelists (#28302)
mistyping in one of the highlighting examples comment -> content (#28139)
Documents applicability of term query to range type (#28166)
Build: Omit dependency licenses check for elasticsearch deps (#28304)
Clean up commits when global checkpoint advanced (#28140)
Implement socket and server ChannelContexts (#28275)
Plugins: Fix meta plugins to install bundled plugins with their real name (#28285)
Build: Fix meta plugin integ test installation (#28286)
Modify Abstract transport tests to use impls (#28270)
Fork Groovy compiler onto compile Java home
[Docs] Update tophits-aggregation.asciidoc (#28273)
Docs: match between snippet to its description (#28296)
[TEST] fix RequestTests#testSearch in case search source is not set
REST high-level client: remove index suffix from indices client method names (#28263)
Fix simple_query_string on invalid input (#28219)
...
This commit is related to #27260. Currently have a channel context that
implements reading and writing logic for socket channels. Additionally,
we have exception contexts to handle exceptions. And accepting contexts
to handle accepted channels. This PR introduces a ChannelContext that
handles close and exception handling for all channel types.
Additionally, it has implementers that provide specific functionality
for socket channels (read and writing). And specific functionality for
server channels (accepting).
There a number of tests in `AbstractSimpleTransportTestCase` that
create `MockTcpTransport` impls. This commit modifies two of these tests
to use the transport implementation that is being tested.
This commit is related to #27260. Right now we have separate read and
write contexts for implementing specific protocol logic. However, some
protocols require a closer relationship between read and write
operations than is allowed by our current model. An example is HTTP
which might require a write if some problem with request parsing was
encountered.
Additionally, some protocols require close messages to be sent when a
channel is shutdown. This is also problematic in our current model,
where we assume that channels should simply be queued for close and
forgotten.
This commit transitions to a single ChannelContext which implements
all read, write, and close logic for protocols. It is the job of the
context to tell the selector when to close the channel. A channel can
still be manually queued for close with a selector. This is how server
channels are closed for now. And this route allows timeout mechanisms on
normal channel closes to be implemented.
This logging message adds considerable noise to many REST tests, if you
are using something like HTTP basic auth in every API call or set any custom
header.
The log level moves from info to debug, so can still be seen if wanted.
* es/master: (30 commits)
[Docs] Fix Java Api index administration usage (#28133)
Fix eclipse build. (#28236)
Never return null from Strings.tokenizeToStringArray (#28224)
Fallback to TransportMasterNodeAction for cluster health retries (#28195)
[Docs] Changes to ingest.asciidoc (#28212)
TEST: Update logging for testAckedIndexing
[GEO] Add WKT Support to GeoBoundingBoxQueryBuilder
Painless: Add whitelist extensions (#28161)
Fix daitch_mokotoff phonetic filter to use the dedicated Lucene filter (#28225)
Avoid doing redundant work when checking for self references. (#26927)
Fix casts in HotThreads. (#27578)
Ignore the `-snapshot` suffix when comparing the Lucene version in the build and the docs. (#27927)
Allow update of `eager_global_ordinals` on `_parent`. (#28014)
Fix NPE on composite aggregation with sub-aggregations that need scores (#28129)
`MockTcpTransport` to connect asynchronously (#28203)
Fix synonym phrase query expansion for cross_fields parsing (#28045)
Introduce elasticsearch-core jar (#28191)
#28218: Update the Lucene version for 6.2.0 after backport
upgrade to lucene 7.2.1 (#28218)
[Docs] Fix an error in painless-types.asciidoc (#28221)
...
The composite aggregation defers the collection of sub-aggregations to a second pass that visits documents only if they
appear in the top buckets. Though the scorer for sub-aggregations is not set on this second pass and generates an NPE if any sub-aggregation
tries to access the score. This change creates a scorer for the second pass and makes sure that sub-aggs can use it safely to check the score of
the collected documents.
The method `initiateChannel` on `TcpTransport` is explicit in that
channels can be connect asynchronously. All production implementations
do connect asynchronously. Only the blocking `MockTcpTransport`
connects in a synchronous manner. This avoids testing some of the
blocking code in `TcpTransport` that waits on connections to complete.
Additionally, it requires a more extensive method signature than
required for other transports.
This commit modifies the `MockTcpTransport` to make these connections
asynchronously on a different thread. Additionally, it simplifies that
`initiateChannel` method signature.
This is related to #27933. It introduces a jar named elasticsearch-core
in the lib directory. This commit moves the JarHell class from server to
elasticsearch-core. Additionally, PathUtils and some of Loggers are
moved as JarHell depends on them.
* master:
Fix lock accounting in releasable lock
Add ability to associate an ID with tasks (#27764)
[DOCS] Removed differencies between text and code (#27993)
text fixes (#28136)
Update getting-started.asciidoc (#28145)
[Docs] Spelling fix in painless-getting-started.asciidoc (#28187)
Fixed the cat.health REST test to accept 4ms, not just 4.0ms (#28186)
Do not keep 5.x commits once having 6.x commits (#28188)
* master: (43 commits)
Rename core module to server (#28180)
upgraded jna from 4.4.0-1 to 4.5.1 (#28183)
[TEST] Do not call RandomizedTest.scaledRandomIntBetween from multiple threads
Primary send safe commit in file-based recovery (#28038)
[Docs] Correct response json in rank-eval.asciidoc
Add scroll parameter to _reindex API (#28041)
Include all sentences smaller than fragment_size in the unified highlighter (#28132)
Modifies the JavaAPI docs related to AggregationBuilder
[Docs] Improvements in script-fields.asciidoc (#28174)
[Docs] Remove Kerberos/SPNEGO Shield plugin (#28019)
Ignore null value for range field (#27845) (#28116)
Fix environment variable substitutions in list setting (#28106)
docs: Replaces indexed script java api docs with stored script api docs
test: ensure we endup with a single segment
Make sure that we don't detect files as maven coordinate when installing a plugin (#28163)
[Tests] temporary disable meta plugin rest tests #28163
meta-plugin should install bin and config at the top level (#28162)
Painless: Add public member read/write access test. (#28156)
Docs: Clarify password protection support with keystore (#28157)
[Docs] fix plugin properties inclusion for plugins authors
...
Today a primary shard transfers the most recent commit point to a
replica shard in a file-based recovery. However, the most recent commit
may not be a "safe" commit; this causes a replica shard not having a
safe commit point until it can retain a safe commit by itself.
This commits collapses the snapshot deletion policy into the combined
deletion policy and modifies the peer recovery source to send a safe
commit.
Relates #10708
We set the watermarks to low values in other test cases to prevent test
failures on nodes with low disk space (if the disk space is too low, the
test will fail anyway but we should not prematurely fail). This commit
sets the watermarks in the single-node test cases to avoid test failures
in such situations.
Relates #28134
This commit adds the ability to package multiple plugins in a single zip.
The zip file for a meta plugin must contains the following structure:
|____elasticsearch/
| |____ <plugin1> <-- The plugin files for plugin1 (the content of the elastisearch directory)
| |____ <plugin2> <-- The plugin files for plugin2
| |____ meta-plugin-descriptor.properties <-- example contents below
The meta plugin properties descriptor is mandatory and must contain the following properties:
description: simple summary of the meta plugin.
name: the meta plugin name
The installation process installs each plugin in a sub-folder inside the meta plugin directory.
The example above would create the following structure in the plugins directory:
|_____ plugins
| |____ <name_of_the_meta_plugin>
| | |____ meta-plugin-descriptor.properties
| | |____ <plugin1>
| | |____ <plugin2>
If the sub plugins contain a config or a bin directory, they are copied in a sub folder inside the meta plugin config/bin directory.
|_____ config
| |____ <name_of_the_meta_plugin>
| | |____ <plugin1>
| | |____ <plugin2>
|_____ bin
| |____ <name_of_the_meta_plugin>
| | |____ <plugin1>
| | |____ <plugin2>
The sub-plugins are loaded at startup like normal plugins with the same restrictions; they have a separate class loader and a sub-plugin
cannot have the same name than another plugin (or a sub-plugin inside another meta plugin).
It is also not possible to remove a sub-plugin inside a meta plugin, only full removal of the meta plugin is allowed.
Closes#27316
This commit changes IndexShardSnapshotStatus so that the Stage is updated
coherently with any required information. It also provides a asCopy()
method that returns the status of a IndexShardSnapshotStatus at a given
point in time, ensuring that all information are coherent.
Closes#26480
Several responses include the shards_acknowledged flag (indicating whether the
requisite number of shard copies started before the completion of the operation)
and there are two different getters used : isShardsAcknowledged() and isShardsAcked().
This PR deprecates the isShardsAcked() in favour of isShardsAcknowledged() in
CreateIndexResponse, RolloverResponse and CreateIndexClusterStateUpdateResponse.
Closes#27784
* master:
Remove out-of-date projectile file
Remove Gradle cheatsheet
Fix reproduction info to point to Gradle wrapper
Update platforms tests to use Gradle wrapper
Update testing docs to reflect Gradle wrapper
Painless: Modify Loader to Load Classes Directly from Definition (#28088)
Update contributing docs to use the Gradle wrapper
Create nio-transport plugin for NioTransport (#27949)
test: replaced try-catch statements with expectThrows(...)
Add getWarmer and getTranslog method to NodeIndicesStats (#28092)
fix doc mistake
Added ASN support for Ingest GeoIP plugin.
Fix global aggregation that requires breadth first and scores (#27942)
This is related to #27260. This commit moves the NioTransport from
:test:framework to a new nio-transport plugin. Additionally, supporting
tcp decoding classes are moved to this plugin. Generic byte reading and
writing contexts are moved to the nio library.
Additionally, this commit adds a basic MockNioTransport to
:test:framework that is a TcpTransport implementation for testing that
is driven by nio.
* es/master:
Introduce Gradle wrapper
Ignore GIT_COMMIT when calculating commit hash
Re-enable bwc tests after #27881 was backported
Set the elasticsearch-nio codebase for tests (#28067)
This commit sets the elasticsearch-nio code base in the
BootstrapForTesting class. This is necessary as that codebase needs
socket permissions. Setting the codebase manually is necessary as
intellij does not package our internal libraries when running tests.
* es/master: (53 commits)
Bump compat version for local depdendent test to 6.2.0
Pass `java.locale.providers=COMPAT` to Java 9 onwards (#28080)
Allow shrinking of indices from a previous major (#28076)
Remove deprecated exceptions (#28059)
Add Writeable.Reader support to TransportResponseHandler (#28010)
Plugins: Add plugin extension capabilities (#27881)
Fix cluster.routing.allocation.enable and cluster.routing.rebalance.enable casing (#28037)
[Test] Fix scores for dcg in RankEvalRequestIT and RankEvalYamlIT
[Docs] Add note on limitation for significant_text with nested objects (#28052)
Enable convert processor to support Long and Double. (#27957)
Enable Wildfly tests on JDK 9 and JDK 10
[Test] Fix allowed delta for calculated scores in DiscountedCumulativeGainTests
[Test] Mute DiscountedCumulativeGainTests on ARM
Only bind loopback addresses when binding to local
Fix assertion in Wildfly build
Fix typo in comment in Wildfly build
Use ephemeral ports in Wildfly tests
Update fuzzy-query.asciidoc (#28032)
Just another elasticsearch library (#27996)
Disable nio test transport (#28028)
...
Allows TransportResponse objects not to implement Streamable anymore. As an example, I've adapted the response handler for ShardActiveResponse, allowing the fields in that class to become final.
This commit adds the infrastructure to plugin building and loading to
allow one plugin to extend another. That is, one plugin may extend
another by the "parent" plugin allowing itself to be extended through
java SPI. When all plugins extending a plugin are finished loading, the
"parent" plugin has a callback (through the ExtensiblePlugin interface)
allowing it to reload SPI.
This commit also adds an example plugin which uses as-yet implemented
extensibility (adding to the painless whitelist).
This commit disables the nio transport as an option for the test
transport in integration tests. This is because it does not currently
run properly in intellij due to socket permissions. It should be
reenabled once #27881 is merged (and the proper permissions are added).
* Adds task dependenciesInfo to BuildPlugin to generate a CSV file with dependencies information (name,version,url,license)
* Adds `ConcatFilesTask.groovy` to concatenates multiple files into one
* Adds task `:distribution:generateDependenciesReport` to concatenate `dependencies.csv` files into a single file (`es-dependencies.csv` by default)
# Examples:
$ gradle dependenciesInfo :distribution:generateDependenciesReport
## Use `csv` system property to customize the output file path
$ gradle dependenciesInfo :distribution:generateDependenciesReport -Dcsv=/tmp/elasticsearch-dependencies.csv
## When branch is not master, use `build.branch` system property to generate correct licenses URLs
$ gradle dependenciesInfo :distribution:generateDependenciesReport -Dbuild.branch=6.x -Dcsv=/tmp/elasticsearch-dependencies.csv
Today we always recover a primary from the last commit point. However
with a new deletion policy, we keep multiple commit points in the
existing store, thus we have chance to find a good starting commit
point. With a good starting commit point, we may be able to throw away
stale operations. This PR rollbacks a primary to a starting commit then
recovering from translog.
Relates #10708
* es/master: (45 commits)
Adapt scroll rest test after backport. relates #27842
Move early termination based on index sort to TopDocs collector (#27666)
Upgrade beats templates that we use for bwc testing. (#27929)
ingest: upgraded ingest geoip's geoip2's dependencies.
[TEST] logging for update by query test #27820
Add elasticsearch-nio jar for base nio classes (#27801)
Use full profile on JDK 10 builds
Require Gradle 4.3
Enable grok processor to support long, double and boolean (#27896)
Add unreleased v6.1.2 version
TEST: reduce blob size #testExecuteMultipartUpload
Check index under the store metadata lock (#27768)
Fixes DocStats to not report index size < -1 (#27863)
Fixed test to be up to date with the new database files.
Upgrade to Lucene 7.2.0. (#27910)
Disable TestZenDiscovery in cloud providers integrations test
Use `_refresh` to shrink the version map on inactivity (#27918)
Make KeyedLock reentrant (#27920)
ingest: Upgraded the geolite2 databases.
[Test] Fix IndicesClientDocumentationIT (#27899)
...
This is related to #27802. This commit adds a jar called
elasticsearch-nio that contains the base nio classes that will be used
for the tcp nio transport and eventually the http nio transport.
The jar does not depend on elasticsearch:core, so all references to core
have been removed.
Today when we get a metadata snapshot directly from a store directory,
we acquire a metadata lock, then acquire an IndexWriter lock. However,
we create a CheckIndex in IndexShard without acquiring the metadata lock
first. This causes a recovery failed because the IndexWriter lock can be
still held by method snapshotStoreMetadata. This commit makes sure to
create a CheckIndex under the metadata lock.
Closes#24481Closes#27731
Relates #24787
The last operation executed in IndicesClientDocumentationIT.testCreate()
is an asynchronous index creation. Because nothing waits for its
completion, on slow machines the index can sometimes be created after
the testCreate() test is finished, and it can fail the following test.
Closes#27754
When the first parameter of `ESTestCase#randomValueOtherThan` is `null`
then run the supplier until it returns non-`null`. Previously,
`randomValueOtherThan` just ran the supplier one time which was
confusing.
Unexpectedly, it looks like not tests rely on the original `null`
handling.
Closes#27821
We currently have a complicated port assignment scheme to make sure that the nodes span off by the internal test cluster will be assigned fixed port ranges that will also not collide between clusters. The port ranges need to be fixed in advance so that the nodes will be able to find each other via `UnicastZenPing`.
This approach worked well for the last few years but we are now at a point that our testing has grown beyond it and we exceed the 5 reusable ranges per JVM. This means that nodes are not always assigned the first 5 ports in their range which causes cluster formation issues. On top of that, most of the clusters that are span up don't even rely on `UnicastZenPing` but rather `MockZenPings` that uses in memory maps for discovery (with the down side that they are not influenced by network disruption simulations).
This PR changes `InternalTestCluster` to use port 0 as a fixed assignment. This will allow the OS to manage ports and will ensure we don't have collisions. For tests that need to simulate network disruptions (and thus can't use `MockZenPings`), a new `UnicastHostProvider` is introduced that is based on the current state of the test cluster. Since that is only resolved at run time, it is aware of the port assignments of the OS.
Closes#27818Closes#27762
This commit moves GlobalCheckpointTracker from the engine to IndexShard, where it better fits logically: Tracking the global checkpoint based on the local checkpoints of all shards in the replication group is not a property of the engine, but rather a property fulfilled by the current primary shard. The LocalCheckpointTracker on the other hand is driven by the contents of the local translog. By moving GlobalCheckpointTracker to IndexShard, it makes little sense to keep the SequenceNumbersService class around - it would only wrap the LocalCheckpointTracker. This commit therefore removes the class and replaces occurrences of SequenceNumbersService in the engine directly by LocalCheckpointTracker.
AnalysisFactoryTestCase checks that the ES custom token filter multi-term
awareness matches the underlying lucene factory. For the trim filter this
won't be the case until LUCENE-8093 is released in 7.3, so we add a
temporary exclusion
Closes#27310
This commit fixes the version tests for release tests. The problem here
is that during release tests all version should be treated as released
so the assertions must be modified accordingly.
Relates #27815
When snapshotting the primary we capture a lucene commit at an arbitrary moment from a sequence number perspective. This means that it is possible that the commit misses operations and that there is a gap between the local checkpoint in the commit and the maximum sequence number.
When we restore, this will create a primary that "misses" operations and currently will mean that the sequence number system is stuck (i.e., the local checkpoint will be stuck). To fix this we should fill in gaps when we restore, in a similar fashion to normal store recovery.
Currently randomNonNegativeLong() returns 0 half as often as any positive long,
but random number generators are typically expected to return
uniformly-distributed values unless otherwise specified. This fixes this issue
by mapping Long.MIN_VALUE directly onto 0 rather than resampling.
This commit is related to #27260. It adds a base NioGroup for use in
different transports. This class creates and starts the underlying
selectors. Different protocols or transports are established by passing
the ChannelFactory to the bindServerChannel or openChannel
methods. This allows a TcpChannelFactory to be passed which will
create and register channels that support the elasticsearch tcp binary
protocol or a channel factory that will create http channels (or other).
Martijn van Groningen
Nhat Nguyen
Vladimir Dolzhenko
Tim Brooks
David Turner
Colin Goodheart-Smithe
Yannick Welsch
Luca Cavanna
Ryan Ernst
Adrien Grand
Zachary Tong
Van0SS
Nik Everett
Jason Tedor
Julie Tibshirani
Jay Modi
Stéphane Campinas
Tanguy Leroux
Jim Ferenczi
Boaz Leskes
olcbean
Simon Willnauer
Lee Hinman
tomcallahan
Mayya Sharipova
Andy Bristol
Christoph Büscher
Yu
Michael Basnight
markharwood
Alexander Reelsen
kel
Igor Motov
Maxime Gréau
Alan Woodward