As a default in V2, the GeoPointField.stored option was set to true. Since this consumes disk space with no positive benefit the default stored option is being reverted back to false.
This commit restores logging the ShardRouting#shardId at the front of
the log messages in ShardStateAction. The reason for this is so that
shard-level log messages have the format "[component][node][shard]
message".
There are two bugs:
- the 'global_ordinals_low_cardinality' mode requires a fielddata-based impl so
that it can extract the segment to global ordinal mapping
- the 'global_ordinals_hash' mode abusively casts to the values source to a
fielddata-based impl while it is not needed
Closes#14882
This commit modifies the handling of cluster states in
o.e.c.a.s.ShardStateAction so that all necessary state is obtained
externally to the ShardStateAction#shardFailed and
ShardStateAction#shardStarted methods. This refactoring permits the
removal of the ClusterService field from ShardStateAction.
This commit applies a minor code cleanup to
o/e/c/ClusterStateObserver.java. In particular
- employ the diamond operator instead of explicitly specifying a
generic type parameter
- use 'L' instead of 'l' for specifying a long literal
- remove redundant static modifier on a nested interface
- remove redundant public access modifiers on interface methods
- reformat the declaration of the four-argument ChangePredicate#apply
- simplify the bodies of ValidationPredicate#apply
This commit fixes multiField support for GeoPointFieldMapper by passing an externalValueContext to the multiField parser. Unit testing is added for multi field coverage.
The MapperService doesn't currently check the
index.mapper.dynamic setting during index creation,
so indices can be created with dynamic mappings even
if this setting is false. Add a check that throws an
exception in this case. Fixes#15381
DedicatedClusterSnapshotRestoreIT#testRestoreIndexWithMissingShards took ~1.5 min to finish
due to timeouts that are applied if not all shards are allocated. Now that the index that has
unallocated shareds is not refreshed the test is more reasonable and runs in 15 sec
With this commit we check more precisely on the result of a bulk
request. It could either be ok, fail or be rejected due to resource
constraints. Previously, we have relied that by default we never
get rejected.
However, this is a valid condition even when retrying. With this
commit we check that we either retried often enough that we don't
get rejected *and* if we got rejected that we maxed out the number
of specified retries.
When specifying a string field, you can either do:
```
{
"foo": "bar"
}
```
or
```
{
"foo": {
"value": "bar",
"boost": 42
}
}
```
The latter option is now removed.
Closes#15388
Removal of the pattern node.addShard() -> calculate weight -> node.removeShard() which is expensive as, beside map lookups, it invalidates caching of precomputed values in ModelNode and ModelIndex. Replaced by adding an additional parameter to the weight function which accounts for the added / removed shard.
This removes the backward compatibility layer with pre-2.0 indices, notably
the extraction of _id, _routing or _timestamp from the source document when a
path is defined.
Today when dynamically mapping a field that is already defined in another type,
we use the regular dynamic mapping logic and try to copy some settings to avoid
introducing conflicts. However this is quite fragile as we don't deal with every
existing setting. This proposes a different approach that will just reuse the
shared field type.
Close#15568
FunctionScoreQuery should do two things that it doesn't do today:
- propagate the two-phase iterator from the wrapped scorer so that things are
still executed efficiently eg. if a phrase or geo-distance query is wrapped
- filter out docs that don't have a high enough score using two-phase
iteration: this way the score is only checked when everything else matches
While doing these changes, I noticed that minScore was ignored when scores were
not needed and that explain did not take it into account, so I fixed these
issues as well.
This changes a couple of things:
Mappings are truly immutable. Before, each field mapper stored a
MappedFieldTypeReference that was shared across fields that have the same name
across types. This means that a mapping update could have the side-effect of
changing the field type in other types when updateAllTypes is true. This works
differently now: after a mapping update, a new copy of the mappings is created
in such a way that fields across different types have the same MappedFieldType.
See the new Mapper.updateFieldType API which replaces MappedFieldTypeReference.
DocumentMapper is now immutable and MapperService.merge has been refactored in
such a way that if an exception is thrown while eg. lookup structures are being
updated, then the whole mapping update will be aborted. As a consequence,
FieldTypeLookup's checkCompatibility has been folded into copyAndAddAll.
Synchronization was simplified: given that mappings are truly immutable, we
don't need the read/write lock so that no documents can be parsed while a
mapping update is being processed. Document parsing is not performed under a
lock anymore, and mapping merging uses a simple synchronized block.
This adds the required changes/checks so that the build can run on
FreeBSD.
There are a few things that differ between FreeBSD and Linux:
- CPU probes return -1 for CPU usage
- `hot_threads` cannot be supported on FreeBSD
From OpenJDK's `os_bsd.cpp`:
```c++
bool os::is_thread_cpu_time_supported() {
#ifdef __APPLE__
return true;
#else
return false;
#endif
}
```
So this API now returns (for each FreeBSD node):
```
curl -s localhost:9200/_nodes/hot_threads
::: {Devil Hunter Gabriel}{q8OJnKCcQS6EB9fygU4R4g}{127.0.0.1}{127.0.0.1:9300}
hot_threads is not supported on FreeBSD
```
- multicast fails in native `join` method - known bug:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246
Which causes:
```
1> Caused by: java.net.SocketException: Invalid argument
1> at java.net.PlainDatagramSocketImpl.join(Native Method)
1> at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:179)
1> at java.net.MulticastSocket.joinGroup(MulticastSocket.java:323)
1> at org.elasticsearch.plugin.discovery.multicast.MulticastChannel$Plain.buildMulticastSocket(MulticastChannel.java:309)
```
So these tests are skipped on FreeBSD.
Resolves#15562
In this commit we increase the queue size of the bulk pool in
BulkProcessorRetryIT to make it less sensitive.
As this test case should stress the pool so bulk processor needs to
back off but not so much that the backoff policy will give up at
some point (which is a valid condition), we still keep it below the
default queue size of 50.
Some tests, but in particular CodecTests, are slow because they test all
versions that ever existed even though they should only test supported
versions.
Today we throttle recoveries only for incoming recoveries. Nodes that have a lot
of primaries can get overloaded due to too many recoveries. To still keep that at bay
we limit the number of threads that are sending files to the target to overcome this problem.
The right solution here is to also throttle the outgoing recoveries that are today unbounded on
the master and don't start the recovery until we have enough resources on both source and target nodes.
The concurrency aspects of the recovery source also added a lot of complexity and additional threadpools
that are hard to configure. This commit removes the concurrent streamns notion completely and sends files
in the thread that drives the recovery simplifying the recovery code considerably.
Outgoing recoveries are not throttled on the master via a allocation decider.
We added this undocumented realtime setting as backup plan long ago
but to date we haven't had a situation where it was a problem. It's reducing
the number of filehandles in the NRT case dramatically and should always be enabled.
Today the logic to async - commit the translog is in every translog instance
itself. While the setting is a per index setting we manageing it per shard. This
polluts the translog code and can more easily be managed in IndexService.
Today we have two variants of translogs for indexing. We only recommend the buffered
one which also has a 20% advantage in indexing speed. This commit removes the option and defaults
to the buffered case. It also hard-wires the translog buffer to 8kb instead of 64kb. We used to
adjust that buffer based on if the shard is active or not, this code has also been removed and
instead we just keep an 8kb buffer arround.
This commit removes `index.translog.flush_threshold_ops` and `index.translog.disable_flush`
in favor of `index.translog.flush_threshold_size`. The number of operations is meaningless by itself and
can easily be turned into a size value with knowledge of the data. Disabling the flush is only useful in
tests and we can set the size value to a really high value. If users really need to do this they can
also apply a very high value like `1PB`.
DocumentMapperParser has both parse and parseCompressed methods. Except that the
parse methods are ONLY used from the unit tests. This commit removes the parse
method and moves all tests to parseCompressed so that they test more
realistically how mappings are managed.
Then I renamed parseCompressed to parse given that this is the only alternative
anyway.
This change adds AbstractValuesSourceParser which will be the new class used to create ValuesSourceAggregatorFactory objects. AbstractValuesSourceParser parses all the parameters required for ValuesSource and passes to the sub-class to parse any other (implementation specific) parameters. After parsing is complete it will call createFactory on the implementing class to create the AggregatorFactory object and then set the ValuesSource specific parameters before returning it.
ValuesSourceAggregatorFactory also now has setter methods so that it can be used as the 'builder' object in the future.
ValuesSourceConfig is now evaluated in the ValuesSourceAggregatorFactory instead of ValueSourceParser. This means that the ValueSourceParser purely deals with parsing the XContent and the logic requiring access to the mappings etc. on the shard is left to the ValuesSourceAggregatorFactory. This means that, in the future, the parsing logic can be moved to the coordinating node.
Resolves conflicts between parent routing and alias routing with the following rule:
* The parent routing is ignored if there is an alias routing that matches the request.
Closes#3068
With this commit we implement a cancellation policy in
BulkProcessor which is aligned for the sync and the async case
and also document it.
Closes#14833.
Simon Willnauer
Simon Willnauer
Jason Tedor
Nicholas Knize
Adrien Grand
Dave
Jim Ferenczi
Robert Muir
Daniel Mitterdorfer
Robert Muir
Yannick Welsch
Lee Hinman
kwangsik
Jun Ohtani
Michael McCandless
Colin Goodheart-Smithe
Zachary Tong
Ryan Ernst