Now that fine-grained script settings are supported (#10116) we can remove support for the script.disable_dynamic setting.
Same result as `script.disable_dynamic: false` can be obtained as follows:
```
script.inline: on
script.indexed: on
```
An exception is thrown at startup when the old setting is set, so we make sure we tell users they have to change it rather than ignoring the setting.
Closes#10286
The query cache is disabled on dfs_query_then_fetch so we need to enforce
query_then_fetch instead of relying on the randomized search type set by the
test framework.
This commit brings the benefits of the `count` search type to search requests
that have a `size` of 0:
- a single round-trip to shards (no fetch phase)
- ability to use the query cache
Since `count` now provides no benefits over `query_then_fetch`, it has been
deprecated.
Close#7630
Even if there is a background thread that periodically closes search contexts
that seem unused (every minute by default), it is important to close search
contexts as soon as possible in order to not keep unnecessary open files or
to prevent segments from being deleted.
This check would help ensure that refactorings of the SearchContext management
like #9296 are correct.
Adds a getter for the actual netty channel in NettyTransportChannel. The
channel can be used by plugins that need access into netty when processing
requests.
FakeRestRequest is used by a few tests and can also be leveraged by
tests outside of elasticsearch. Moving the package will mean the class
gets exported as part of the test jar.
We already force a refresh in index/create ops, to clear version map
when it's using too much RAM, but we were failing to do this for
deletes, so an app that does tons of deletes with no indexing, and has
set refresh_interval to -1, would have version map using unbounded
RAM.
Closes#10312
After processing mapping updates from the master, we compare the resulting binary representation of them and compare it the one cluster state has. If different, we send a refresh mapping request to master, asking it to reparse the mapping and serialize them again. This mechanism is used to update the mapping after a format change caused by a version upgrade.
The very same process can also be triggered when an old master leaves the cluster, triggering a local cluster state update. If that update contains old mapping format, the local node will again signal the need to refresh, but this time there is no master to accept the request. Instead of failing (which we now do because of #10283, we should just skip the notification and wait for the next elected master to publish a new mapping (triggering another refresh if needed).
Closes#10311
Even if there is a background thread that periodically closes search contexts
that seem unused (every minute by default), it is important to close search
contexts as soon as possible in order to not keep unnecessary open files or
to prevent segments from being deleted.
This check would help ensure that refactorings of the SearchContext management
like #9296 are correct.
When the index service (which holds shards) fails to be created as a result of a shard being allocated on a node, we should fail the relevant shard, otherwise, it will remain stuck.
Same goes when there is a failure to process updated mappings form the master.
Note, both failures typically happen when the node is misconfigured (i.e. missing plugins, ...), since they get created and processed on the master node before being published.
closes#10283
Doc values significantly reduced heap usage, which results in faster
GCs. This change makes the default for doc values dynamic: any
field that is indexed but not analyzed now has doc values. This only
affects fields on indexes created with 2.0+.
closes#8312closes#10209
The local DocumentMapper is updated while parsing and dynamic fields are added before
parsing has finished. If parsing fails after a dynamic field has been added already
then the field was not added to the cluster state but was present in the local mapper of this
node. New documents with the same field would not necessarily cause an update either and
after restarting the node the mapping for these fields were lost. Instead the new fields
should always be updated.
closes#9851closes#9874
We currently have a single bw comp test (FunctionScoreBackwardCompatibilityTests) that requires inline scripts on.
After introducing fine-grained script settings, we moved the internal cluster to use the newer settings, but they are not supported by older nodes started as part of the bw comp tests. Moved script settings out of the default settings, so they won't be part of the ordinary settings when running bw comp tests.
Added logic in FunctionScoreBackwardCompatibilityTests to enable dynamic scripts using the proper setting, depending on the version of the node.
Allow to on/off scripting based on their source (where they get loaded from), the operation that executes them and their language.
The settings cover the following combinations:
- mode: on, off, sandbox
- source: indexed, dynamic, file
- engine: groovy, expressions, mustache, etc
- operation: update, search, aggs, mapping
The following settings are supported for every engine:
script.engine.groovy.indexed.update: sandbox/on/off
script.engine.groovy.indexed.search: sandbox/on/off
script.engine.groovy.indexed.aggs: sandbox/on/off
script.engine.groovy.indexed.mapping: sandbox/on/off
script.engine.groovy.dynamic.update: sandbox/on/off
script.engine.groovy.dynamic.search: sandbox/on/off
script.engine.groovy.dynamic.aggs: sandbox/on/off
script.engine.groovy.dynamic.mapping: sandbox/on/off
script.engine.groovy.file.update: sandbox/on/off
script.engine.groovy.file.search: sandbox/on/off
script.engine.groovy.file.aggs: sandbox/on/off
script.engine.groovy.file.mapping: sandbox/on/off
For ease of use, the following more generic settings are supported too:
script.indexed: sandbox/on/off
script.dynamic: sandbox/on/off
script.file: sandbox/on/off
script.update: sandbox/on/off
script.search: sandbox/on/off
script.aggs: sandbox/on/off
script.mapping: sandbox/on/off
These will be used to calculate the more specific settings, using the stricter setting of each combination. Operation based settings have precedence over conflicting source based ones.
Note that the `mustache` engine is affected by generic settings applied to any language, while native scripts aren't as they are static by definition.
Also, the previous `script.disable_dynamic` setting can now be deprecated.
Closes#6418Closes#10116Closes#10274
In #9893, an enabled flag was added for _field_names. However,
backcompat for indexes created before 1.3.0 (when _field_names
was added) was lost. This change corrects the mapper
to always be disabled when used with older indexes that
cannot have _field_names.
closes#10268
Deleting a type from an index is inherently dangerous because
the type can be recreated with new mappings which may conflict
with existing segments still using the old mappings. This
removes the ability to delete a type (similar to how deleting
fields within a type is not allowed, for the same reason).
closes#8877closes#10231
This adds the Explanation to the explain score again. It is needed
because the explanation of script functions will otherwise not contain
an explanation of _score if boost mode is set to replace.
closes#9826
Fail merge if validate_lat or validate_lon values are not equal. This will prevent inconsistencies between geo_points in a merged index, and parse exceptions for bounding_box and distance filters.
Also merged separate GeoPoint test classes into a single GeoPointFieldMapperTest to be consistent with GeoShapeFieldMapperTests.
closes#10164
* add compiler workarounds for JDK bug JI-9019884
* remove permgen specification during tests (this results in an error on java 9)
* fix threadpool grow/shrink to call methods in the right order (this results in IAE with java 9)
This reverts commit 166fd04239aaa95ed19ad46187deeb8eb0006b15.
Turns out that having log.warn produces a duplicated warn log, as the same message is already logged warn in NettyTranspo
rt#exceptionCaught.
If a request comes in at the same moment the timeout handler for it runs, we may leak a timeoutInfoHolder and erroneously log "Transport response handler not found of id" . The same issue could cause the request tracer to fire a traceUnresolvedResponse call instead of traceReceivedResponse , causing a failure of testTracerLog ( see #10187 ) .
This commit makes sure timeoutInfoHolder is visible before removing the corresponding RequestHolder. It also unifies the TransportService.Adapter#remove(requestId) with TransportService.Adapter#onResponseReceived(requestId), as they are always called together to indicate a response was received.
Closes#10187Closes#10220
When a primary moves to another node, we cancel ongoing recoveries and retry from the primary's new home. At the moment this happens when the primary relocation *starts*. It's a shame as we cancel recoveries that may be close to completion and will finish before the primary has been fully relocated. This commit only triggers the cancelation once the primary relocation is completed.
Next to this, it fixes a race condition between recovery cancellation and the recovery completion. At the moment we may trigger remove a recovered shard just after it was completed. Instead, we should use the recovery cancellation logic to make sure only one code path is followed.
All of the above caused the recoverWhileUnderLoadWithNodeShutdown test to fail (see http://build-us-00.elastic.co/job/es_core_15_debian/32/ ). The test creates an index and then increasingly disallows nodes for it, until only 1 node is left in the allocation filtering rules. Normally, this means we stay in green, but the premature recovery cancellation plus the race condition mentioned above caused a shard to be failed and stay unassigned and the test asserts to fail. This happens due to the following sequence:
- The shard has finished recovering and sent the master a shard started command.
- The recovery is cancelled locally, removing the index shard.
- Master starts shard (deleting it's other copy).
- Local node gets a cluster state with the shard started in it, which cause it to send a shard failed (to make the master aware).
- Shard is failed and can't be re-assigned due to the allocation filter.
The recoverWhileUnderLoadWithNodeShutdown is also adapted a bit to fit the current behavior of allocation filtering (in the past it used to really shut down nodes). Last, all tests in that class are given better names to fit the current terminology.
Clsoes #10218
Today we simply fetch the shards metadata without verifying the
index UUID the shard belongs to. We recently added this UUID
to the shard state metadata. This commit adds verification
to the shard metadata fetching to prevent bringing shards
back into an index it doesn't belong to due to name collisions.
