This commit adds the support for exclusion filter to the response filtering (filter_path) feature. It changes the XContentBuilder APIs so that it now accepts two types of filters: inclusive and exclusive. Filters are no more String arrays but sets of String instead.
Instead of get, set and remove we do get, remove and then set to avoid type conflicts in IngestDocument.
If the set still fails we try to restore the original field in ingest document.
Closes#19892
already has a file of the same name in the Store, but is different
in content (different checksum/length), then those files are first
deleted before restoring the files in question.
This changes Elasticsearch to automatically downgrade `text` and
`keyword` fields into appropriate `string` fields when changing the
mapping of indexes imported from 2.x. This allows users to use the
modern, documented syntax against 2.x indexes. It also makes it clear
that reindexing in order to recreate the index in 5.0 is required for
any long lived indexes. This change is useful for the times when you
can't (cluster is just starting, not stable enough for reindex) or
shouldn't (index will only live 90 days or something).
Now that #20081 is merged we can check that cacheKey is consistent across equal search requests, something that wasn't true before due to ordering of map keys when using index boost.
Relates to #19986
Removed null check for token, if we are outside the null it already means it is null.
Fixed typo in comment and remove leftover assignment to unused local variable.
Netty3/4 TcpTransport implementations are creating thread factories with a "http_server" thread prefix whereas it should start with "transport_server" and let the "http_server" prefix for the HttpServerTransport implementations.
This test is periodically failing. As I suspect that the GCDisruption scheme is somehow making the wrong node block on
its cluster state update thread, I've added some more logging and a thread dump once the given assertion triggers
again.
today we fsync in a blocking fashion where all threads block while another
syncs. Yet, we can improve this and make use of the async infrastrucutre added
for `wait_for_refresh` and make fsyncing single threaded while all other threads
can continue indexing. The syncing thread then notifies a listener once the requests
location is synced. This also allows to send docs to replicas before its actually fsynced
allowing for cocurrent replica processing.
This patch has a significant impact on performance on slower discs. An initial single node benchmark
shows that on very fast SSDs there is no noticable impact but on slow spinning disk this
patch shows a ~32% performance improvement.
```
NVME SSD:
336ec0ac9a (master):
Total docs/sec: 47200.9
Total docs/sec: 46440.4
23543a97e3e7f72a31e26b50e00931919784426c (async wait for translog):
Total docs/sec: 47461.6
Total docs/sec: 46188.3
-------------------------------------------------------------------
Spinning disk:
336ec0ac9a (master):
Total docs/sec: 22733.0
Total docs/sec: 24129.8
23543a97e3e7f72a31e26b50e00931919784426c (async wait for translog):
Total docs/sec: 32724.1
Total docs/sec: 32845.4
--------------------------------------------------------------------
```
Currently, after at least one pipeline is registered it is getting rebuilt on every single cluster state update, even when this update is not related to ingest metadata. This change adds a check that the ingest metadata changed before trying to rebuild all pipelines.
Adds an explicit recoverySource field to ShardRouting that characterizes the type of recovery to perform:
- fresh empty shard copy
- existing local shard copy
- recover from peer (primary)
- recover from snapshot
- recover from other local shards on same node (shrink index action)
* Fix NPE during search with source filtering if the source is disabled.
Instead of throwing an NPE, a search response with source filtering will not contain the source if it is disabled in the mapping.
Closes#7758
* Created unit tests for FetchSourceSubPhase. Tests similar to SourceFetchingIT.
Removed SourceFetchingIT#testSourceDisabled (now covered via unit test FetchSourceSubPhaseTests#testSourceDisabled).
* Updated FetchSouceSubPhase unit tests per comments.
Renamed main unit test method.
Use assertEquals and assertNull instead of assertThat (less code).
Fixes an issue where the value for the `script.engine.<lang>.inline`
settings would be _set_ properly, but would not accurately be reflected
in the `include_defaults` output. Adds a test to ensure the default raw
setting is now correct.
Resolves#20159
* Make sure indexBoost is serialized in a consistent order
* remove hasIndexBoost by using indexBoost size
* Make sure phrase suggester's collateParams is serialized in consistent
order
* Make StreamOutput writer to serialize maps in consistent order
The class Setting holds a static reference to a deprecation logger
instance. When the class initializer for Setting runs, it starts
triggering log4j initialization. There is a chain of initializations
from InternalSettingsPreparer to Environment to Setting that triggers
this initialization before log4j configuration has occurred. This commit
modifies this initialization so that initialization is not done eagerly.
Relates #20170
This makes GET operations more consistent with `_search` operations which expect
`(stored_)fields` to work on stored fields and source filtering to work on the
`_source` field. This is now possible thanks to the fact that GET operations
do not read from the translog anymore (#20102) and also allows to get rid of
`FieldMapper#isGenerated`.
The `_termvectors` API (and thus more_like_this too) was relying on the fact
that GET operations would extract fields from either stored fields or the source
so the logic to do this that used to exist in `ShardGetService` has been moved
to `TermVectorsService`. It would be nice that term vectors do not rely on this,
but this does not seem to be a low hanging fruit.
Non-stale shard copies are currently tracked using their allocation ids in the cluster state. When a node leaves the cluster, shard copies of that node are marked as stale by removing their allocation ids from the active set in the cluster. For full cluster restarts, this can have the unwanted effect that only the last node holding a copy of the shard will be seen as non-stale. The other shard copies are not really stale though as long as no writes have happened on this shard copy. Shard copies should thus only be marked as stale (by the master in the cluster state) if other active shards have received writes.
This commit implements the above logic and also renames the persistent structure used to track non-stale shard copies from "active_allocations" to "in_sync_allocations" as we now also support tracking non-stale shard copies that have no active routing entries in the cluster state.
We need to get the string representation of numbers in order to include in
`_all`. However this has a cost and disabling `_all` is rather common so we
should look into skipping it.
The network types in use on a cluster can be useful information to have,
so this commit adds aggregate metrics for the network types in use in a
cluster to the cluster stats.
Relates #20144
This moves the Writer interface from StreamOutput into Writeable, as a peer of its inner Reader interface. This should hopefully help to avoid random functional interfaces being created for the same purpose.
It also makes use of the moved class by updating writeMapOfLists and readMapOfLists.
This removes final from the RestClient, Response, and Sniffer classes so that outside code can mock them. Their constructors are already package private, so there's not much that can go wrong.
Jim Ferenczi
Tanguy Leroux
Martijn van Groningen
Ali Beyad
Areek Zillur
Nik Everett
javanna
Greg Marzouka
markwalkom
Munish Goyal
Yannick Welsch
Jun Ohtani
Simon Willnauer
Simon Willnauer
Igor Motov
qwerty4030
Lee Hinman
Nicolas Ruflin
Josh Becker
Xiang Chen
Chris Earle
Jason Tedor
Adrien Grand
Daniel Mitterdorfer
Daniel Mitterdorfer
Sarwar Bhuiyan