After another round of input from @cbuescher this adds a few more sanity
checks to request parsing.
In addition adds (back) support for the reverse option.
Today index names are often resolved lazily, only when they are really
needed. This can be problematic especially when it gets to mapping updates
etc. when a node sends a mapping update to the master but while the request
is in-flight the index changes for whatever reason we would still apply the update
since we use the name of the index to identify the index in the clusterstate.
The problem is that index names can be reused which happens in practice and sometimes
even in a automated way rendering this problem as realistic.
In this change we resolve the index including it's UUID as early as possible in places
where changes to the clusterstate are possible. For instance mapping updates on a node use a
concrete index rather than it's name and the master will fail the mapping update iff
the index can't be found by it's <name, uuid> tuple.
Closes#17048
Changes:
- no more option to configure eager/lazy loading of the norms (useless now
that orms are disk-based)
- only the `string`, `text` and `keyword` fields support the `norms` setting
- the `norms` setting takes a boolean that decides whether norms should be
stored in the index but old options are still supported to give users time
to upgrade
- setting a `boost` no longer implicitely enables norms (for new indices only,
this is still needed for old indices)
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.
Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
To make API's output more easy to read we are suppressing stack traces (#12991) unless explicitly requested by setting `error_trace=true` on the request. To compensate we are logging the stacktrace into the logs so people can look it up even the error_trace wasn't enabled. Currently we do so using the `INFO` level which can be verbose if an api is called repeatedly by some automation. For example, if someone tries to read from an index that doesn't exist we will respond with a 404 exception and log under info every time. We should reduce the level to `DEBUG` as we do with other API driven errors. Internal errors (rest codes >=500) are logged as WARN.
Closes#16627
This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.
Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.
Relates to #14406Closes#17072
When unzipping a plugin zip, the zip entries are resolved relative to
the directory being unzipped into. However, there are currently no
checks that the entry name was not absolute, or relatively points
outside of the plugin dir. This change adds a check for those two cases.
This adds methods and tests to ScriptSortBuilder that
makes it implement NamedWritable and adds the fromXContent
method needed to read itseld from xContent.
Test failures showed problems with passing down
the same collate parameter map reference from the
phrase suggestion builder to the context where.
This changes the collate parameter setters to
make a shallow copy of the map passed in.
This commit adds fields bytes_recovered and files_recovered to the cat
recovery API. These fields, respectively, indicate the total number of
bytes and files recovered. Additionally, for consistency, some totals
fields and translog recovery fields have been renamed.
Closes#17064
This tries to remove friction to upgrade to 5.0 that would be caused by mapping
changes:
- old ways to specify mapping settings (eg. store: yes instead of store:true)
will still work but a deprecation warning will be logged
- string mappings that only use the most common options will be upgraded
automatically to text/keyword
The change to move dynamic mapping handling to the end of document
parsing has an edge case which can cause dynamic mappings to fail
document parsing. If field a.b is added as an as part of the root update,
followed by a.c.d, then we need to expand the mappers on the stack,
since a is hidden inside the root update which exists on the stack.
This change adds a test for this case, as well as tries to better
document how the logic works for building up the stack before adding a
dynamic mapper.
Sequence of events that lead to the NPE:
- avg metric returns NaN for buckets
- Movavg skips NaN or null buckets, and simply re-uses the existing bucket (e.g. doesn't add
a 'movavg' field)
- Derivative references Movavg, the bucket resolution returns null because Movavg wasn't added
to the bucket, NPE when trying to subtract null values
The ingest stats include the following statistics:
* `ingest.total.count`- The total number of document ingested during the lifetime of this node
* `ingest.total.time_in_millis` - The total time spent on ingest preprocessing documents during the lifetime of this node
* `ingest.total.current` - The total number of documents currently being ingested.
* `ingest.total.failed` - The total number ingest preprocessing operations failed during the lifetime of this node
Also these stats are returned on a per pipeline basis.
Currently, the cluster service is tightly coupled to the transport service by both managing node connections and requiring the bound address in order to create the local disco node. This commit introduces a new NodeConnectionsService which is in charge of node connection management and makes it possible to remove all network related calls from the cluster service. The local DiscoNode is now created by DiscoveryNodeService and is set both the cluster service and the transport service during node start up.
Closes#16788Closes#16872
Currently all SortBuilder implementations have their separate order
field. This PR moves this up to SortBuilder, together with setter
and getter and makes sure the default is set to SortOrder.ASC
except for `_score` sorting where the default is SortOrder.DESC.
This commit updates the documentation for GeoPointField by removing all references to the coerce and doc_values parameters. DocValues are enabled in lucene GeoPointField by default (required for boundary filtering). The QueryBuilders are updated to automatically normalize points (ignoring the coerce parameter) for any index created onOrAfter version 2.2.
Use index UUID to lookup indices on IndicesService
Today we use the index name to lookup index instances on the IndicesService
which applied to search requests but also to index deletion etc. This commit
moves the interface to expect an Index instance which is a tuple
and looks up the index by uuid rather than by name. This prevents accidental modification
of the wrong index if and index is recreated or searching from the wrong index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.
Today we use the index name to lookup index instances on the IndicesService
which applied to search reqeusts but also to index deletion etc. This commit
moves the interface to expcet and `Index` instance which is a <name, uuid> tuple
and looks up the index by uuid rather than by name. This prevents accidential modificaiton
of the wrong index if and index is recreated or searching from the _wrong_ index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.
Closes#17001
Occasionally the .geohash suffix in Geo{Distance|DistanceRange}Query would conflict with a mapping that defines a sub-field by the same name. This occurs often with nested and multi-fields a mapping defines a geo_point sub-field using the field name "geohash". Since the QueryParser already handles parsing geohash encoded geopoints without requiring the ".geohash" suffix, the suffix parsing can be removed altogether.
This commit removes the .geohash suffix parsing, adds explicit test coverage for the nested query use-case, and adds random distance queries to the nested query test suite.
In particular, this test ensures we don't restart the master node until
we know the index deletion has taken effect on master and the master
eligible nodes.
Closes#16917Closes#16890
This change makes ScoreSortBuilder implement NamedWriteable, adds
equals() and hashCode() and also implements parsing ScoreSortBuilder
back from xContent. This is needed for the ongoing Search refactoring.
This was lost in refactoring even on the 2.x branch. The slow-log
is not per index not per shard anymore such that we don't add the
shard ID as the logger prefix. This commit adds back the index
name as part of the logging message not as a prefix on the logger
for better testabilitly.
Closes#17025
_wait_for_completion defaults to false. If set to true then the API will
wait for all the tasks that it finds to stop running before returning. You
can use the timeout parameter to prevent it from waiting forever. If you
don't set a timeout parameter it'll default to 30 seconds.
Also adds a log message to rest tests if any tasks overrun the test. This
is just a log (instead of failing the test) because lots of tasks are run
by the cluster on its own and they shouldn't cause the test to fail. Things
like fetching disk usage from the other nodes, for example.
Switches the request to getter/setter style methods as we're going that
way in the Elasticsearch code base. Reindex is all getter/setter style.
Closes#16906
Currently the message stays in the `UnassignedInfo` for the shard,
however, it would be very useful to know the exact point (time-wise)
that the cancellation happened when diagnosing an issue.
Relates to debugging #16357
Apparently lucene6 is way more picky with respect to corrupting files
that are not fsynced that's why this test sometimes failed after the lucene6
upgrade.
This commit reduces the maximum number of threads required in the
bootstrap check. This limit can be reduced since the generic thread pool
is no longer unbounded.
Relates #17003
The generic thread pool was previously configured to be able to create
an unlimited number of threads. The thinking is that tasks that are
submitted to its work queue must execute and should not block waiting
for a worker. However, in cases of heavy load, this can lead to an
explosion in the number of threads; this can even lead to a feedback
loop that exacerbates the problem. What is more, this can even bump into
OS limits on the number of threads that can be created.
This commit limits the number of threads in the generic thread pool to
four times the bounded number of processors.
Relates #17003
This merge commit merges #16682 which adds the ability to define a index-global
default similarity but at the same time prevents overriding built-in similarities
for new indices. Old indices where able to do this int the past which should not
be punished even though this is not possible anymore.
Closes#16594Closes#16682
This removes the need for accessing the SearchContext when parsing Sort elements
to queries. After applying the patch only a QueryShardContext is needed.
Relates to #15178
Move some test methods from AnalylzeActionIT to RestAnalyzeActionTest
Allow string explain param if it can parse
Fix wrong param name in rest-api-spec
Closes#16925
We removed leniencey from version parsing which caught problems with
-SNAPSHOT suffixes on plugin properies. This commit removes the -SNAPSHOT
from both es and the extension version and adds tests to ensure we can
parse older versions that allowed -SNAPSHOT in BWC way.
Elasticsearch 5.0 will come with alpha versions which is not supported
in the current version scheme. This commit adds support for aplpha starting
with es 5.0.0 in a backwards compatible way.
Date range aggregations used to be unable to use a `time_zone` parameter to e.g. be applied
int date math roundings like in `now/d` (see #10130 as an example). After the aggregation
refactoring, the time_zone parameter has been pulled up to ValuesSourceAggregatorBuilder
and can now be used in date range aggregations as well.
This change adds randomized time zone settings to the existing IT tests to verify that the
`time_zone` parameter is honored when calculating the bucket boundaries. Also moving
the DateRangeTests from module-groovy/messy back to core as DateRangeIT, sharing
common script mocks with DateHistogramIT and adding documentation for the
`time_zone` parameter in the date range aggregation docs.
Closes#10130
Internally the put pipeline API uses this information in node info API to validate if all specified processors in a pipeline exist on all nodes in the cluster.
Areek Zillur
Ali Beyad
Simon Willnauer
Christoph Büscher
Isabel Drost-Fromm
Adrien Grand
Jason Tedor
Boaz Leskes
David Pilato
Yannick Welsch
Ryan Ernst
Zachary Tong
Zachary Tong
Martijn van Groningen
Isabel Drost-Fromm
Nik Everett
Nicholas Knize
Simon Willnauer
Alexander Kazakov
Lee Hinman
Jun Ohtani
Robert Muir
Robert Muir