Merge feature/ingest branch into master branch.
This adds the ingest feature to ES that allows to preprocess document before indexing on an ingest node.
By default a node is an ingest node. Documents are preprocessed via a pipeline. A pipeline consists
out of one or more processors Each processor makes one or more modifications to a document processed.
There are many types of processors available out-of-the-box that are designed to make a specific change to a document being processed. In a cluster many pipeline can be configured via dedicated pipeline APIs. An new option on the bulk
and index APIs allows to control what pipeline is picked for preprocessing. If no pipeline is specified then the ingest
feature is skipped and no preprocessing takes place.
Boolean parsing is now strict. Also added isIngestNode methods to DiscoveryNode to align this setting with the existing node.data node.master and node.client. Removed NodeModule#isIngestEnabled methods that are not needed anymore.
This commit removes non-reproducible randomness from
SearchWhileCreatingIndexIT. The cause of the non-reproducible randomness
is the use of a random draws for the shard preference inside of a
non-deterministic while loop. Because the inner while loop executed a
non-deterministic number of times, the draws for the next iteration of
the outer loop would be impacted by this making the random draws
non-reproducible. The solution is to move the random draws outside of
the while loop (just make a single draw for the prefernce and increment
with a counter), and remove the outer loop iteration instead using an
annotation to get the desired repetitions.
Closes#16208
This commit amends the logging statement in two places in
ShardStateAction to log the full shard ID instead of just the numerical
shard ID (without the index name).
This commit migrates all the settings under network service to the new settings infra.
It also adds some chaining utils to make fall back settings slightly less verbose.
Breaking (but I think acceptable) - network.tcp.no_delay and network.tcp.keep_alive used to accept the value `default` which make us not set them at all on netty. Our default was true so we weren't using this feature. I removed it and now we only accept a true boolean.
* `test.cluster.node.seed` was only used in one place where it wasn't adding value and was now replaced with a constant
* `tests.portsfile` is now an official setting and has been renamed to `node.portsfile`
this commit also convert `search.default_keep_alive` and `search.keep_alive_interval` to the new settings infrastrucutre
This commit also uses a try/finally with success pattern instead of catching
and excpetion. TranslogTests reproduce with `-Dtests.seed=DF6A38BAE739227A` every
time.
Closes#16142
TranslogWriter.closeIntoReader transfers the file ownership from a writer to a reader and closes the writer. If the transfer fails, we need to make sure we closed the underlying channel as the writer is already closes.
See: http://build-us-00.elastic.co/job/es_core_master_regression/4355
Relates to #16142
Today we run the metadata upgrade only on the current major version
but this should run on every upgrade at least once to ensure we don't miss
an important check or upgrade.
This commit fixes a minor issue with the shard ID that is logged while
indexing in DiscoveryWithServiceDisruptionsIT#testAckedIndexing. The
issue is that the operation routing hash could lead to a negative
remainder modulo the number of primaries (if the hash itself is
negative) but should instead be the normalized positive remainder. This
issue only impacts the logging of the shard ID as the actual shard ID
used during indexing is computed elsewhere but would cause the shard ID
in the affected logging statement to not match shard IDs that are logged
elsewhere.
this change allows us to open existing IndexMetaData that contains invalid, removed settings
or settings with invalid values and instead of filling up the users disks with exceptions we _archive_
the settings with and `archive.` prefix. This allows us to warn the user via logs (once it's archived) as
well as via external tools like the upgrade validation tool since those archived settings will be preserved
even over restarts etc. It will prevent indices from failing during the allocaiton phase but instead will
print a prominent warning on index metadata recovery from disk.
Affects match, multi_match, query_string and simple_query_string queries.
Direct bool queries are not affected anymore (minimum_should_match is applied even if the coord factor is disabled).
Colin Goodheart-Smithe
Martijn van Groningen
Isabel Drost-Fromm
Isabel Drost-Fromm
javanna
Jason Tedor
Ryan Ernst
Boaz Leskes
Tanguy Leroux
markharwood
Simon Willnauer
Yannick Welsch
Simon Willnauer
Daniel Mitterdorfer
Jim Ferenczi
Adrien Grand
Tal Levy
Clinton Gormley