This is related to #27260 and #28898. This commit adds the transport-nio
plugin as a random option when running the http smoke tests. As part of
this PR, I identified an issue where cors support was not properly
enabled causing these tests to fail when using transport-nio. This
commit also fixes that issue.
Adds support for `ignore_unmapped` parameter in geo distance sorting,
which is functionally equivalent to specifying an `unmapped_type` in
the field sort.
Closes#28152
Several AcknowledgedResponse implementations only parse the boolean acknowledged
flag and then create an instance of their class using that flag. This can be
simplified by adding this basic parser to the superclass, provide a common
helper method and call the appropriate ctor in the fromXContent methods.
This change moves an integration test that relies on setting
the value of a static variable (boolean max clause count) to
an unit test where we are sure that the same jvm is used to access
the static variable.
By default span_multi query will limit term expansions = boolean max clause.
This will limit high heap usage in case of high cardinality term
expansions. This applies only if top_terms_N is not used in inner multi
query.
Today if a user omits the `_source` entirely or modifies the source
on indexing we have no chance to re-create the document after it has
been added. This is an issue for CCR and recovery based on soft deletes
which we are going to make the default. This change adds an additional
recovery source if the source is disabled or modified that is only kept
around until the document leaves the retention policy window.
This change adds a merge policy that efficiently removes this extra source
on merge for all document that are live and not in the retention policy window
anymore.
This is related to #28898. This commit adds the acceptor thread name to
the method checking if this thread is a transport thread. Additionally,
it modifies the nio http transport to use the same worker name as the
netty4 http server transport.
This is related to #27260. This commit combines the AcceptingSelector
and SocketSelector classes into a single NioSelector. This change
allows the same selector to handle both server and socket channels. This
is valuable as we do not necessarily want a dedicated thread running for
accepting channels.
With this change, this commit removes the configuration for dedicated
accepting selectors for the normal transport class. The accepting
workload for new node connections is likely low, meaning that there is
no need to dedicate a thread to this process.
The primary shard copy on the following has authority of the replication
operations that occur on the following side in cross-cluster
replication. Yet today we are using the primary term directly from the
operations on the leader side. Instead we should be replacing the
primary term on the following side with the primary term of the primary
on the following side. This commit does this by copying the translog
operations with the corrected primary term. This ensures that we use
this primary term while applying the operations on the primary, and when
replicating them across to the replica (where the replica request was
carrying the primary term of the primary shard copy on the follower).
Currently the engine is initialized with a hardcoded 256MB of RAM. Elasticsearch
may never use more than that for a given shard, `IndexingMemoryController` only
has the power to flush segments to disk earlier in case multiple shards are
actively indexing and use too much memory.
While this amount of memory is enough for an index with few fields and larger
RAM buffers are not expected to improve indexing speed, this might actually be
little for an index that has many fields.
Kudos to @bleskes for finding it out when looking into a user who was reporting
a **much** slower indexing speed when upgrading from 2.x to 5.6 with an index
that has about 20,000 fields.
We currently have a specific REST action to retrieve all indices and types mappings, which used internally the get index API. This doesn't seem to be required anymore though as the existing RestGetMappingAction could as well take the requests with no indices and types specified.
This commit removes the RestGetAllMappingsAction in favour of using RestGetMappingAction also for requests that don't specify indices nor types.
The Index Audit trail allows the override of the template index
settings with settings specified on the conf file.
A bug will manifest when such conf file settings are specified
for templates that need to be upgraded. The bug is an endless
upgrade loop because the upgrade, although successful, is
not reckoned as such by the upgrade service.
move `finger_print`, `pattern` and `standard_html_strip` analyzers
to analysis-common module. (both AnalysisProvider and PreBuiltAnalyzerProvider)
Changed PreBuiltAnalyzerProviderFactory to extend from PreConfiguredAnalysisComponent and
changed to make sure that predefined analyzers are always instantiated with the current
ES version and if an instance is requested for a different version then delegate to PreBuiltCache.
This is similar to the behaviour that exists today in AnalysisRegistry.PreBuiltAnalysis and
PreBuiltAnalyzerProviderFactory. (#31095)
Relates to #23658
* elastic/master:
[DOCS] Creates rest-api folder in docs
[Rollup] Disallow index patterns that match the rollup index (#30491)
Add cors support to NioHttpServerTransport (#30827)
[DOCS] Fixes security example (#31082)
Allow terms query in _rollup_search (#30973)
This PR adapts/utilizes recent enhancements in Lucene-7.4:
- Replaces exactNumDocs by the soft-deletes count in SegmentCommitInfo.
This enhancement allows us to back out changes introduced in #30228.
- Always configure the soft-deletes field in IWC
This snapshot includes:
- LUCENE-8341: Record soft deletes in SegmentCommitInfo which will resolve#30851
- LUCENE-8335: Enforce soft-deletes field up-front
This is related to #28898. This commit adds cors support to the nio http
transport. Most of the work is copied directly from the netty module
implementation. Additionally, this commit adds tests for the nio http
channel.
* master:
Removing erroneous repeat
Adapt bwc versions after backporting #30983 to 6.4
[Tests] Muting RatedRequestsTests#testXContentParsingIsNotLenient
TEST: Retry synced-flush if ongoing ops on primary (#30978)
Fix docs build.
Only auto-update license signature if all nodes ready (#30859)
Add BlobContainer.writeBlobAtomic() (#30902)
Add a doc value format to binary fields. (#30860)
When the last indexing operation is completed, we will fire a global
checkpoint sync. Since a global checkpoint sync request is a replication
request, it will acquire an index shard permit on the primary when
executing. If this happens at the same time while we are issuing the
synced-flush, the synced-flush request will fail as it thinks there are
in-flight operations. We can avoid such situation by retrying another
synced-flush if the current request fails due to ongoing operations on
the primary.
Closes#29392
This commit adds a new writeBlobAtomic() method to the BlobContainer
interface that can be implemented by repository implementations which
support atomic writes operations.
When the BlobContainer implementation does not provide a specific
implementation of writeBlobAtomic(), then the writeBlob() method is used.
Related to #30680
This will be necessary for the `docvalue_fields` option to work correctly once
we use the field's doc-value format to format doc-value fields. Binary values
are formatted as base64-encoded strings.
* es/master:
Take into account the return value of TcpTransport.readMessageLength(...) in Netty4SizeHeaderFrameDecoder
Move caching of the size of a directory to `StoreDirectory`. (#30581)
Clarify docs about boolean operator precedence. (#30808)
Docs: remove notes on sparsity. (#30905)
Fix MatchPhrasePrefixQueryBuilderTests#testPhraseOnFieldWithNoTerms
run overflow forecast a 2nd time as regression test for elastic/ml-cpp#110 (#30969)
Improve documentation of dynamic mappings. (#30952)
Decouple MultiValueMode. (#31075)
Docs: Clarify constraints on scripted similarities. (#31076)
Update get.asciidoc (#31084)
In spite of the existing caching, I have seen a number of nodes hot threads
where one thread had been spending all its cpu on computing the size of a
directory. I am proposing to move the computation of the size of the directory
to `StoreDirectory` in order to skip recomputing the size of the directory if
no changes have been made. This should help with users that have read-only
indices, which is very common for time-based indices.
Currently this class takes care of moth selecting the relevant value, and
replacing missing values if any. This is fine for sorting, which always needs
to do both at the same time, but we also have a number of aggregations and
script utils that need to retain information about missing values so this change
proposes to decouple selection of the relevant value and replacement of missing
values.
* master:
Add get mappings support to high-level rest client (#30889)
Fix index prefixes to work with span_multi (#31066)
[DOCS] Removes redundant authorization pages
[DOCS] Re-adds custom realm
Change ObjectParser exception (#31030)
Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (#31073)
* Fix index prefixes to work with span_multi
Text fields that use `index_prefixes` can rewrite `prefix` queries into
`term` queries internally. This commit fix the handling of this rewriting
in the `span_multi` query.
This change also copies the index options of the text field into the
prefix field in order to be able to run positional queries. This is mandatory
for `span_multi` to work but this could also be useful to optimize `match_phrase_prefix`
queries in a follow up. Note that this change can only be done on indices created
after 6.3 since we set the index options to doc only in this version.
Fixes#31056
* master:
Match phrase queries against non-indexed fields should throw an exception (#31060)
In the internal highlighter APIs, use the field type as opposed to the mapper. (#31039)
[DOCS] Removes duplicated authentication pages
Enable customizing REST tests blacklist (#31074)
Make sure KeywordFieldMapper#clone preserves split_queries_on_whitespace. (#31049)
[DOCS] Moves machine learning overview to stack-docs
[ML] Add secondary sort to ML events (#31063)
[Rollup] Specialize validation exception for easier management (#30339)
Adapt bwc versions after backporting #31045 to 6.3
Remove usage of explicit type in docs (#29667)
Share common readFrom/writeTo code in AcknowledgeResponse (#30983)
Adapt bwc versions after backporting #31045 to 6.x
Mute MatchPhrase*QueryBuilderTests
[Docs] Fix typo in watcher conditions documentation (#30989)
Remove wrong link in index phrases doc
Move pipeline APIs to ingest namespace (#31027)
[DOCS] Fixes accounting setting names (#30863)
[DOCS] Rewords _field_names documentation (#31029)
Index phrases (#30450)
Remove leftover debugging from PTCMDT
Fix PTCMDT#testMinVersionSerialization
Make Persistent Tasks implementations version and feature aware (#31045)
ObjectParser should throw XContentParseExceptions, not IAE. A dedicated parsing
exception can includes the place where the error occurred.
Closes#30605
This snapshot includes:
- LUCENE-8341: Record soft deletes in SegmentCommitInfo which will resolve#30851
- LUCENE-8335: Enforce soft-deletes field up-front
When `lenient=false`, attempts to create match phrase queries with custom analyzers against non-text fields will throw an IllegalArgumentException.
Also changes `*Match*QueryBuilderTests` so that it avoids this scenario
Fixes#31061
The majority of Responses inheriting from AcknowledgeResponse implement
the readFrom and writeTo serialization method in the same way. Moving this
as a default into AcknowledgeResponse and letting the few exceptions that
need a slightly different implementation handle this themselves saves a lot
of duplication.
Specifying `index_phrases: true` on a text field mapping will add a subsidiary
[field]._index_phrase field, indexing two-term shingles from the parent field.
The parent analysis chain is re-used, wrapped with a FixedShingleFilter.
At query time, if a phrase match query is executed, the mapping will redirect it
to run against the subsidiary field.
This should trade faster phrase querying for a larger index and longer indexing
times.
Relates to #27049
This commit fixes an issue with
PersistentTasksCustomMetaDataTests#testMinVersionSerialization. There
were two problems here:
- some versions do not have future compatible version (e.g., betas)
- the feature logic was incorrect
