Merge feature/ingest branch into master branch.
This adds the ingest feature to ES that allows to preprocess document before indexing on an ingest node.
By default a node is an ingest node. Documents are preprocessed via a pipeline. A pipeline consists
out of one or more processors Each processor makes one or more modifications to a document processed.
There are many types of processors available out-of-the-box that are designed to make a specific change to a document being processed. In a cluster many pipeline can be configured via dedicated pipeline APIs. An new option on the bulk
and index APIs allows to control what pipeline is picked for preprocessing. If no pipeline is specified then the ingest
feature is skipped and no preprocessing takes place.
- move ingest plugin docs to core reference docs
- move geoip processor docs to plugins/ingest-geoip.asciidoc
- add missing options tables for some processors
- add description of pipeline definition
- add description of processor definitions including common parameters
like "tag" and "on_failure"
Boolean parsing is now strict. Also added isIngestNode methods to DiscoveryNode to align this setting with the existing node.data node.master and node.client. Removed NodeModule#isIngestEnabled methods that are not needed anymore.
This commit removes non-reproducible randomness from
SearchWhileCreatingIndexIT. The cause of the non-reproducible randomness
is the use of a random draws for the shard preference inside of a
non-deterministic while loop. Because the inner while loop executed a
non-deterministic number of times, the draws for the next iteration of
the outer loop would be impacted by this making the random draws
non-reproducible. The solution is to move the random draws outside of
the while loop (just make a single draw for the prefernce and increment
with a counter), and remove the outer loop iteration instead using an
annotation to get the desired repetitions.
Closes#16208
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
This adds support for returning the size of an array
or an collection, in addition to access fields via
`array.0` you can specify `array.size` to get its size.
This commit amends the logging statement in two places in
ShardStateAction to log the full shard ID instead of just the numerical
shard ID (without the index name).
This commit migrates all the settings under network service to the new settings infra.
It also adds some chaining utils to make fall back settings slightly less verbose.
Breaking (but I think acceptable) - network.tcp.no_delay and network.tcp.keep_alive used to accept the value `default` which make us not set them at all on netty. Our default was true so we weren't using this feature. I removed it and now we only accept a true boolean.
* `test.cluster.node.seed` was only used in one place where it wasn't adding value and was now replaced with a constant
* `tests.portsfile` is now an official setting and has been renamed to `node.portsfile`
this commit also convert `search.default_keep_alive` and `search.keep_alive_interval` to the new settings infrastrucutre
Simon Willnauer
Clinton Gormley
Martijn van Groningen
Isabel Drost-Fromm
Isabel Drost-Fromm
Tal Levy
javanna
Luca Cavanna
Jason Tedor
David Pilato
Alexander Reelsen
Ryan Ernst
Boaz Leskes
Tanguy Leroux
markharwood
Simon Willnauer
Yannick Welsch
Daniel Mitterdorfer
Jim Ferenczi
Kevin Adams
Adrien Grand
Glen Smith