* master:
Painless: Simplify Naming in Lookup Package (#32177)
Handle missing values in painless (#32207)
add support for write index resolution when creating/updating documents (#31520)
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
Remove indication of future multi-homing support (#32187)
Rest test - allow for snapshots to take 0 milliseconds
Make x-pack-core generate a pom file
Rest HL client: Add put watch action (#32026)
Build: Remove pom generation for plugin zip files (#32180)
Fix comments causing errors with Java 11
Fix rollup on date fields that don't support epoch_millis (#31890)
Detect and prevent configuration that triggers a Gradle bug (#31912)
[test] port linux package packaging tests (#31943)
Revert "Introduce a Hashing Processor (#31087)" (#32178)
Remove empty @return from JavaDoc
Adjust SSLDriver behavior for JDK11 changes (#32145)
[test] use randomized runner in packaging tests (#32109)
Add support for field aliases. (#32172)
Painless: Fix caching bug and clean up addPainlessClass. (#32142)
Call setReferences() on custom referring tokenfilters in _analyze (#32157)
Fix BwC Tests looking for UUID Pre 6.4 (#32158)
Improve docs for search preferences (#32159)
use before instead of onOrBefore
Add more contexts to painless execute api (#30511)
Add EC2 credential test for repository-s3 (#31918)
A replica can be promoted and started in one cluster state update (#32042)
Fix Java 11 javadoc compile problem
Fix CP for namingConventions when gradle home has spaces (#31914)
Fix `range` queries on `_type` field for singe type indices (#31756)
[DOCS] Update TLS on Docker for 6.3 (#32114)
ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are
Remove versionType from translog (#31945)
Switch distribution to new style Requests (#30595)
Build: Skip jar tests if jar disabled
Painless: Add PainlessClassBuilder (#32141)
Build: Make additional test deps of check (#32015)
Disable C2 from using AVX-512 on JDK 10 (#32138)
Build: Move shadow customizations into common code (#32014)
Painless: Fix Bug with Duplicate PainlessClasses (#32110)
Remove empty @param from Javadoc
Re-disable packaging tests on suse boxes
Docs: Fix missing example script quote (#32010)
[ML] Wait for aliases in multi-node tests (#32086)
[ML] Move analyzer dependencies out of categorization config (#32123)
Ensure to release translog snapshot in primary-replica resync (#32045)
Handle TokenizerFactory TODOs (#32063)
Relax TermVectors API to work with textual fields other than TextFieldType (#31915)
Updates the build to gradle 4.9 (#32087)
Mute :qa:mixed-cluster indices.stats/10_index/Index - all’
Check that client methods match API defined in the REST spec (#31825)
Enable testing in FIPS140 JVM (#31666)
Fix put mappings java API documentation (#31955)
Add exclusion option to `keep_types` token filter (#32012)
[Test] Modify assert statement for ssl handshake (#32072)
* master:
Add get field mappings to High Level REST API Client (#31423)
[DOCS] Updates Watcher examples for code testing (#31152)
TEST: Add bwc recovery tests with synced-flush index
[DOCS] Move sql to docs (#31474)
[DOCS] Move monitoring to docs folder (#31477)
Core: Combine doExecute methods in TransportAction (#31517)
IndexShard should not return null stats (#31528)
fix repository update with the same settings but different type (#31458)
Fix Mockito trying to mock IOException that isn't thrown by method (#31433) (#31527)
Node selector per client rather than per request (#31471)
Core: Combine messageRecieved methods in TransportRequestHandler (#31519)
Upgrade to Lucene 7.4.0. (#31529)
[ML] Add ML filter update API (#31437)
Allow multiple unicast host providers (#31509)
Avoid deprecation warning when running the ML datafeed extractor. (#31463)
REST high-level client: add simulate pipeline API (#31158)
Get Mapping API to honour allow_no_indices and ignore_unavailable (#31507)
[PkiRealm] Invalidate cache on role mappings change (#31510)
[Security] Check auth scheme case insensitively (#31490)
In NumberFieldType equals and hashCode, make sure that NumberType is taken into account. (#31514)
[DOCS] Fix REST tests in SQL docs
[DOCS] Add code snippet testing in more ML APIs (#31339)
Core: Remove ThreadPool from base TransportAction (#31492)
[DOCS] Remove fixed file from build.gradle
Rename createNewTranslog to fileBasedRecovery (#31508)
Test: Skip assertion on windows
[DOCS] Creates field and document level security overview (#30937)
[DOCS] Significantly improve SQL docs
[DOCS] Move migration APIs to docs (#31473)
Core: Convert TransportAction.execute uses to client calls (#31487)
Return transport addresses from UnicastHostsProvider (#31426)
Ensure local addresses aren't null (#31440)
Remove unused generic type for client execute method (#31444)
Introduce http and tcp server channels (#31446)
* master:
Upgrade to Lucene-7.4.0-snapshot-518d303506 (#31360)
Rankeval: Fold template test project into main module (#31203)
Add QA project and fixture based test for discovery-ec2 plugin (#31107)
[Docs] Remove reference to repository-s3 plugin creating an S3 bucket (#31359)
REST Client: NodeSelector for node attributes (#31296)
LLClient: Fix assertion on windows
Add details section for dcg ranking metric (#31177)
[ML] Re-enable tests muted in #30982
This snapshot includes:
- LUCENE-8341: Record soft deletes in SegmentCommitInfo which will resolve#30851
- LUCENE-8335: Enforce soft-deletes field up-front
* master:
Add get mappings support to high-level rest client (#30889)
Fix index prefixes to work with span_multi (#31066)
[DOCS] Removes redundant authorization pages
[DOCS] Re-adds custom realm
Change ObjectParser exception (#31030)
Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (#31073)
This snapshot includes:
- LUCENE-8341: Record soft deletes in SegmentCommitInfo which will resolve#30851
- LUCENE-8335: Enforce soft-deletes field up-front
* master:
Avoid randomization bug in FeatureAwareTests
Adjust BWC version on client features
Add TRACE, CONNECT, and PATCH http methods (#31035)
Adjust BWC version on client features
[DOCS] Make geoshape docs less memory hungry (#31014)
Fix handling of percent-encoded spaces in Windows batch files (#31034)
[Docs] Fix a typo in Create Index naming limitation (#30891)
Introduce client feature tracking (#31020)
Ensure that index_prefixes settings cannot be changed (#30967)
REST high-level client: add delete ingest pipeline API (#30865)
[ML][TEST] Fix bucket count assertion in all tests in ModelPlotsIT (#31026)
Allow rollup job creation only if cluster is x-pack ready (#30963)
Fix interoperability with < 6.3 transport clients (#30971)
Add an option to split keyword field on whitespace at query time (#30691)
[Tests] Fix alias names in PutIndexTemplateRequestTests (#30960)
REST high-level client: add get ingest pipeline API (#30847)
Cross Cluster Search: preserve remote status code (#30976)
High-level client: list tasks failure to not lose nodeId (#31001)
[DOCS] Fixes links (#31011)
Watcher: Give test a little more time
Reuse expiration date of trial licenses (#30950)
Remove unused query methods from MappedFieldType. (#30987)
Transport client: Don't validate node in handshake (#30737)
[DOCS] Clarify not all PKCS12 usable as truststores (#30750)
HLRest: Allow caller to set per request options (#30490)
Remove version read/write logic in Verify Response (#30879)
[DOCS] Update readme for testing x-pack code snippets (#30696)
Ensure intended key is selected in SamlAuthenticatorTests (#30993)
Core: Remove RequestBuilder from Action (#30966)
* es/master: (24 commits)
Add missing_bucket option in the composite agg (#29465)
Rename index_prefix to index_prefixes (#30932)
Rename methods in PersistentTasksService (#30837)
[DOCS] Fix watcher file location
Update the version checks around range bucket keys, now that the change was backported.
Use dedicated ML APIs in tests (#30941)
[DOCS] Remove reference to platinum Docker image (#30916)
Minor clean-up in InternalRange. (#30886)
stable filemode for zip distributions (#30854)
[DOCS] Adds missing TLS settings for auditing (#30822)
[test] packaging: use shell when running commands (#30852)
Fix location of AbstractHttpServerTransport (#30888)
[test] packaging test logging for suse distros
Moved keyword tokenizer to analysis-common module (#30642)
Upgrade to Lucene-7.4-snapshot-1cbadda4d3 (#30928)
Limit the scope of BouncyCastle dependency (#30358)
[DOCS] Reset edit links (#30909)
Fix IndexTemplateMetaData parsing from xContent (#30917)
Remove log traces in AzureStorageServiceImpl and fix test (#30924)
Deprecate accepting malformed requests in stored script API (#28939)
...
* master:
QA: Add xpack tests to rolling upgrade (#30795)
Modify state of VerifyRepositoryResponse for bwc (#30762)
Reduce CLI scripts to one-liners on Windows (#30772)
Simplify number of shards setting (#30783)
Replace Request#setHeaders with addHeader (#30588)
[TEST] remove endless wait in RestClientTests (#30776)
[Docs] Fix script-fields snippet execution (#30693)
Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 (#30778)
[DOCS] Add SAML configuration information (#30548)
[DOCS] Remove X-Pack references from SQL CLI (#30694)
Make http pipelining support mandatory (#30695)
[Docs] Fix typo in circuit breaker docs (#29659)
[Feature] Adding a char_group tokenizer (#24186)
[Docs] Fix broken cross link in documentation
Test: wait for netty threads in a JUnit ClassRule (#30763)
Increase the maximum number of filters that may be in the cache. (#30655)
[Security] Include an empty json object in an json array when FLS filters out all fields (#30709)
[TEST] Wait for CS to be fully applied in testDeleteCreateInOneBulk
Add more yaml tests for get alias API (#29513)
Ignore empty completion input (#30713)
[DOCS] fixed incorrect default
[ML] Filter undefined job groups from update calendar actions (#30757)
Fix docs failure on language analyzers (#30722)
[Docs] Fix inconsistencies in snapshot/restore doc (#30480)
Enable installing plugins from snapshots.elastic.co (#30765)
Remove fedora 26, add 28 (#30683)
Accept Gradle build scan agreement (#30645)
Remove logging from elasticsearch-nio jar (#30761)
Add Delete Repository High Level REST API (#30666)
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.
Closes#30741Closes#30577
* master:
Scripting: Remove getDate methods from ScriptDocValues (#30690)
Upgrade to Lucene-7.4.0-snapshot-59f2b7aec2 (#30726)
[Docs] Fix single page :docs:check invocation (#30725)
Docs: Add uptasticsearch to list of clients (#30738)
[DOCS] Removes out-dated x-pack/docs/en/index.asciidoc
[DOCS] Removes redundant index.asciidoc files (#30707)
[TEST] Reduce forecast overflow to disk test memory limit (#30727)
Plugins: Remove meta plugins (#30670)
[DOCS] Moves X-Pack configurationg pages in table of contents (#30702)
TEST: Add engine log to testCorruptFileThenSnapshotAndRestore
[ML][TEST] Fix bucket count assertion in ModelPlotsIT (#30717)
[ML][TEST] Make AutodetectMemoryLimitIT less fragile (#30716)
Default copy settings to true and deprecate on the REST layer (#30598)
[Build] Add test admin when starting gradle run with trial license and
This implementation lazily (on 1st forecast request) checks for available diskspace and creates a subfolder for storing data outside of Lucene indexes, but as part of the ES data paths.
Tests: Fail if test watches could not be triggered (#30392)
[ML] add version information in case of crash of native ML process (#30674)
Make TransportClusterStateAction abide to our style (#30697)
Change required version for Get Settings transport API changes to 6.4.0 (#30706)
[DOCS] Fixes edit URLs for stack overview (#30583)
Silence sleep based watcher test
[TEST] Adjust version skips for movavg/movfn tests
[DOCS] Replace X-Pack terms with attributes
[ML] Clean left behind model state docs (#30659)
Correct typos
filters agg docs duplicated 'bucket' word removal (#30677)
top_hits doc example description update (#30676)
[Docs] Replace InetSocketTransportAddress with TransportAdress (#30673)
[TEST] Account for increase in ML C++ memory usage (#30675)
User proper write-once semantics for GCS repository (#30438)
Remove bogus file accidentally added
Add detailed assert message to IndexAuditUpgradeIT (#30669)
Adjust fast forward for token expiration test (#30668)
Improve explanation in rescore (#30629)
Deprecate `nGram` and `edgeNGram` names for ngram filters (#30209)
Watcher: Fix watch history template for dynamic slack attachments (#30172)
Fix _cluster/state to always return cluster_uuid (#30656)
[Tests] Add debug information to CorruptedFileIT
# Conflicts:
# test/framework/src/main/java/org/elasticsearch/indices/analysis/AnalysisFactoryTestCase.java
* master:
Upgrade to Lucene-7.4-snapshot-6705632810 (#30519)
add version compatibility from 6.4.0 after backport, see #30319 (#30390)
Security: Simplify security index listeners (#30466)
Add proper longitude validation in geo_polygon_query (#30497)
Remove Discovery.AckListener.onTimeout() (#30514)
Build: move generated-resources to build (#30366)
Reindex: Fold "with all deps" project into reindex (#30154)
Isolate REST client single host tests (#30504)
Solve Gradle deprecation warnings around shadowJar (#30483)
SAML: Process only signed data (#30420)
Remove BWC repository test (#30500)
Build: Remove xpack specific run task (#30487)
AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests
LLClient: Add setJsonEntity (#30447)
Expose CommonStatsFlags directly in IndicesStatsRequest. (#30163)
Silence IndexUpgradeIT test failures. (#30430)
Bump Gradle heap to 1792m (#30484)
[docs] add warning for read-write indices in force merge documentation (#28869)
Avoid deadlocks in cache (#30461)
Test: remove hardcoded list of unconfigured ciphers (#30367)
mute SplitIndexIT due to https://github.com/elastic/elasticsearch/issues/30416
Docs: Test examples that recreate lang analyzers (#29535)
BulkProcessor to retry based on status code (#29329)
Add GET Repository High Level REST API (#30362)
add a comment explaining the need for RetryOnReplicaException on missing mappings
Add `coordinating_only` node selector (#30313)
Stop forking groovyc (#30471)
Avoid setting connection request timeout (#30384)
Use date format in `date_range` mapping before fallback to default (#29310)
Watcher: Increase HttpClient parallel sent requests (#30130)
# Conflicts:
# x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/LocalStateCompositeXPackPlugin.java
* master:
Set the new lucene version for 6.4.0
[ML][TEST] Clean up jobs in ModelPlotIT
Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
Watcher: Ensure trigger service pauses execution (#30363)
[DOCS] Added coming qualifiers in changelog
[DOCS] Commented out empty sections in the changelog to fix the doc build. (#30372)
Security: reduce garbage during index resolution (#30180)
Make RepositoriesMetaData contents unmodifiable (#30361)
Change quad tree max levels to 29. Closes#21191 (#29663)
Test: use trial license in qa tests with security
[ML] Add integration test for model plots (#30359)
SQL: Fix bug caused by empty composites (#30343)
[ML] Account for gaps in data counts after job is reopened (#30294)
InternalEngineTests.testConcurrentOutOfOrderDocsOnReplica should use two documents (#30121)
Change signature of Get Repositories Response (#30333)
Tests: Use different watch ids per test in smoke test (#30331)
[Docs] Add term query with normalizer example
Adds Eclipse config for xpack licence headers (#30299)
Watcher: Make start/stop cycle more predictable and synchronous (#30118)
[test] add debug logging for packaging test
[DOCS] Removed X-Pack Breaking Changes
[DOCS] Fixes link to TLS LDAP info
Update versions for start_trial after backport (#30218)
Packaging: Set elasticsearch user to have non-existent homedir (#29007)
[DOCS] Fixes broken links to bootstrap user (#30349)
Fix NPE when CumulativeSum agg encounters null/empty bucket (#29641)
Make licensing FIPS-140 compliant (#30251)
[DOCS] Reorganizes authentication details in Stack Overview (#30280)
Network: Remove http.enabled setting (#29601)
Fix merging logic of Suggester Options (#29514)
[DOCS] Adds LDAP realm configuration details (#30214)
[DOCS] Adds native realm configuration details (#30215)
ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled (#30316)
[DOCS] Enables edit links for X-Pack pages (#30278)
Packaging: Unmark systemd service file as a config file (#29004)
SQL: Reduce number of ranges generated for comparisons (#30267)
Tests: Simplify VersionUtils released version splitting (#30322)
Cancelling a peer recovery on the source can leak a primary permit (#30318)
Added changelog entry for deb prerelease version change (#30184)
Convert server javadoc to html5 (#30279)
Create default ES_TMPDIR on Windows (#30325)
[Docs] Clarify `fuzzy_like_this` redirect (#30183)
Post backport of #29658.
Fix docs of the `_ignored` meta field.
Remove MapperService#types(). (#29617)
Remove useless version checks in REST tests. (#30165)
Add a new `_ignored` meta field. (#29658)
Move repository-azure fixture test to QA project (#30253)
# Conflicts:
# buildSrc/version.properties
# server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
Upgrade to lucene-7.4.0-snapshot-1ed95c097b
This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
This *mostly* silences `javadoc`'s warning about defaulting to
generating html4 files by enabling generating html5 file for the
projects for which that works. It didn't work in a half dozen projects,
about half of which I've fixed in this PR, entirely by replacing
`<tt>thing</tt>` with `{@code thing}`.
There are a few remaining projects that contain javadoc with invalid
html5. I'll fix those projects in a followup.
Some features have been deprecated since `6.0` like the `_parent` field or the
ability to have multiple types per index. This allows to remove quite some
code, which in-turn will hopefully make it easier to proceed with the removal
of types.
* Decouple XContentBuilder from BytesReference
This commit removes all mentions of `BytesReference` from `XContentBuilder`.
This is needed so that we can completely decouple the XContent code and move it
into its own dependency.
While this change appears large, it is due to two main changes, moving
`.bytes()` and `.string()` out of XContentBuilder itself into static methods
`BytesReference.bytes` and `Strings.toString` respectively. The rest of the
change is code reacting to these changes (the majority of it in tests).
Relates to #28504
Using custom rules in the icu_collation filter can fail on Windows. If the rules are interpreted
as a file location, this leads to an InvalidPathException when trying to read the rules from a file.
This new snapshot mostly brings a change to TopFieldCollector which can now
early terminate collection when trackTotalHits is `false`.
As a follow-up, we should replace our usage of
`EarlyTerminatingSortingCollector` with this new option.
The main highlight of this new snapshot is that it introduces the opportunity
for queries to opt out of caching. In case a query opts out of caching, not only
will it never be cached, but also no compound query that wraps it will be
cached.
* Enhances exists queries to reduce need for `_field_names`
Before this change we wrote the name all the fields in a document to a `_field_names` field and then implemented exists queries as a term query on this field. The problem with this approach is that it bloats the index and also affects indexing performance.
This change adds a new method `existsQuery()` to `MappedFieldType` which is implemented by each sub-class. For most field types if doc values are available a `DocValuesFieldExistsQuery` is used, falling back to using `_field_names` if doc values are disabled. Note that only fields where no doc values are available are written to `_field_names`.
Closes#26770
* Addresses review comments
* Addresses more review comments
* implements existsQuery explicitly on every mapper
* Reinstates ability to perform term query on `_field_names`
* Added bwc depending on index created version
* Review Comments
* Skips tests that are not supported in 6.1.0
These values will need to be changed after backporting this PR to 6.x
Today we return a `String[]` that requires copying values for every
access. Yet, we already store the setting as a list so we can also directly
return the unmodifiable list directly. This makes list / array access in settings
a much cheaper operation especially if lists are large.
This change adds a fromXContent method to Settings that allows to read
the xcontent that is produced by toXContent. It also replaces the entire settings
loader infrastructure and removes the structured map representation. Future PRs will
also tackle the `getAsMap` that exposes the internal represenation of settings for
better encapsulation.
Nhat Nguyen
Armin Braun
Adrien Grand
Julie Tibshirani
Martijn van Groningen
Jim Ferenczi
Nik Everett
Alan Woodward
Lee Hinman
Henrik Lindström
Colin Goodheart-Smithe
Simon Willnauer
Md. Abdulla-Al-Sun
Jason Tedor