Once we delete the the index on a node we are closing all resources
and subsequently need to delete all shards contents from disk. Yet
this happens today under a lock (the shard lock) that needs to be
acquried in order to execute any operation on the shards data
path. We try to delete all the index meta-data once we acquired
all the shard lock but this operation can run into a timeout which causes
the index to remain on disk. Further, all shard data will be left on
disk if the timeout is reached.
This commit removes all the shards data just before the shard lock
is release as the last operation on a shard that belongs to a deleted
index.
additional element per segment.
This commit adds a verbose flag to the _segments api. Currently the
only additional information returned when set to true is the full
ram tree from lucene for each segment.
This is our only cache which is not 'exact' and might allow for stalled results.
Additionally, a similar cache that we have and needs to perform lookups in other
indices in order to run queries is the script index, and for this index we rely
on the filesystem cache, so we should probably do the same with terms filters
lookups.
Close#9056
The `cluster.routing.allocation.balance.primary` setting has caused
a lot of confusion in the past while it has very little benefit form a
shard allocatioon point of view. Users tend to modify this value to
evently distribute primaries across the nodes which is dangerous since
a prmiary flag on it's own can trigger relocations. The primary flag for a shard
is should not have any impact on cluster performance unless the high level feature
suffereing from primary hotspots is buggy. Yet, this setting was intended to be a
tie-breaker which is not necessary anymore since the algorithm is deterministic.
This commit removes this setting entriely.
Today this not populated (null) in these cases. But it would be useful to have
this available, even just for improved error messages.
The trickiest part today is the handling of 1.2.x files written with
lucene 4.8+ which have both ES checksums and lucene ones. This is now simplified:
when the lucene checksum is there, we always use it.
Closes#9152
In some situations the shard balanceing weight delta becomes negative. Yet,
a negative delta is always treated as `well balanced` which is wrong. I wasn't
able to reproduce the issue in any way other than useing the real world data
from issue #9023. This commit adds a fix for absolute deltas as well as a base
test class that allows to build tests or simulations from the cat API output.
Closes#9023
If someone blocks on it and it is interrupted, we throw an ElasticsearchIllegalStateException. We should not set Thread.currentThread().interrupt(); in this case because we already communicate the interrupt through an exception.
Similar to #9001Closes#9141
A couple of changes that triggerred a refactoring in Elasticsearch:
- LUCENE-6148: Accountable.getChildResources returns a collection instead of
a list.
- LUCENE-6121: CachingTokenFilter now propagates reset(), as a result
SimpleQueryParser.newPossiblyAnalyzedQuery has been fixed to not reset both
the underlying stream and the wrapper (otherwise lucene would barf because of
a doubl reset).
- LUCENE-6119: The auto-throttle issue changed a couple of method
names/parameters. It also made
`UpdateSettingsTests.testUpdateMergeMaxThreadCount` dead slow so I muted this
test until we clea up merge throttling to use LUCENE-6119.
Close#9145
In the case you try to merge two settings, one being an array and one being
a field, together, the settings were merged instead of being overridden.
First config
my.value: 1
Second config
my.value: [ 2, 3 ]
If you execute
settingsBuilder().put(settings1).put(settings2).build()
now only values 2,3 will be in the final settings
Closes#8381
Plugin installation failed when bin/, conf/ and plugins/ directories are on different file systems. The method File.move() can't be used to move a non-empty directory between different filesystems.
I didn't find a simple way to unittest that, even with in-memory filesystems like jimfs or the Lucene test framework.
Closes#8999
If bulk index request fails due to a disconnect, unavailable shard etc, the request is
retried once before actually failing. However, even in case of failure the documents
might already be indexed. For autogenerated ids the request must not add the
documents again and therfore canHaveDuplicates must be set to true.
closes#8788
A recent situation occured where a MultiPolygon coordinate array was accidentally defined as a single polygon with multiple holes. Since the intent was a MultiPolygon, the holes of the unintended Polygon fell outside the outer shell. This exposed a bug in the contains logic inside BasePolygonBuilder. An ArrayIndexOutOfBoundsException was being thrown instead of a more useful ElasticsearchParseException( "hole is not within polygon" ). This pull request fixes the bug and adds additional unit tests for verifying proper MultiPolygon type parsing.
closes#9071
if we reopen an index and the majority of the replicas where
not created the reopen will fail sicne on master this runs with
local gatway all the time.
This can lead to sporadic shard creating timeouts if the same shard is
created, closed and created again on the same node. The reason for this is
that we holding on to the store reference while blocking on the mapping update
that will prevent the shard lock from being released. Holding the lock is unnecessary
in this case and can simply be removed.
ShapeBuilder expected coordinates for Envelope types in strict Top-Left, Bottom-Right order. Given that GeoJSON does not enforce coordinate order (as seen in #8672) clients could specify envelope bounds in any order and be compliant with the GeoJSON spec but not the ES ShapeBuilder logic. This change loosens the ShapeBuilder requirements on envelope coordinate order, reordering where necessary.
closes#2544closes#9067closes#9079closes#9080
Cleans up the testReusePeerRecovery test as well
The actual fix is in TransportNodesListShardStoreMetaData.java, which
needs to use `nodeEnv.shardDataPaths` instead of `nodeEnv.shardPaths`.
Due to the difficulty in tracking this down, I've added a lot of
additional logging. This also fixes a logging issue in GatewayAllocator
This allows specifying the path an index will be at.
`index.data_path` is specified in the settings when creating an index,
and can not be dynamically changed.
An example request would look like:
POST /myindex
{
"settings": {
"number_of_shards": 2,
"data_path": "/tmp/myindex"
}
}
And would put data in /tmp/myindex/0/index/0 and /tmp/myindex/0/index/1
Since this can be used to write data to arbitrary locations on disk, it
requires enabling the `node.enable_custom_paths` setting in
elasticsearch.yml on all nodes.
Relates to #8976
The terms filter lookup mechanism today caches filters. Because of this, the
cache values depend on two things: the values that can be found in the lookup
index AND the mapping of the local index, since changing the mapping can change
the way that the filter is parsed. We should make the cache depend solely on
the content of the lookup index.
For instance the issue I was seeing was due to the following scenario:
- create index1 with _id indexed
- run terms filter with lookup, the parsed filter looks like `_id: 1 OR _id: 2`
- remove index1
- create index1 with _id not indexed
- run terms filter without lookup, the parsed filter is `_uid: type#1 OR _uid: type#2` (the _id field mapper knows how to use the _uid field when _id is not indexed)
- run terms filter with lookup, the filter is fetched from the cache: `_id: 1 OR _id: 2` but does not match anything since `_id` is not indexed.
Close#9027
Running a terms filter on a single term is equivalent to loading a postings
list into a bit set and then returning the bit set instead of reading the
postings list on the fly.
Close#9014
PR #8978 included 4 unnecessary enumeration values ('cw', 'clockwise', 'ccw', 'counterclockwise'). Since the ShapeBuilder.parse method handles these as strings and maps them to LEFT and RIGHT enumerators, respectively, their enumeration counterpart is unnecessary. This minor change adds 4 static convenience variables (COUNTER_CLOCKWISE, CLOCKWISE, CCW, CW) for purposes of the API and removes the unnecessary values from the Orientation Enum.
closes#9035
This feature adds an optional orientation parameter to the GeoJSON document and geo_shape mapping enabling users to explicitly define how they want Elasticsearch to interpret vertex ordering. The default uses the right-hand rule (counterclockwise for outer ring, clockwise for inner ring) complying with OGC Simple Feature Access standards. The parameter can be explicitly specified for an entire index using the geo_shape mapping by adding "orientation":{"left"|"right"|"cw"|"ccw"|"clockwise"|"counterclockwise"} and/or overridden on each insert by adding the same parameter to the GeoJSON document.
closes#8764
Simon Willnauer
Ryan Ernst
Adrien Grand
Robert Muir
Martijn van Groningen
Boaz Leskes
Alexander Reelsen
tlrx
Clinton Gormley
Britta Weber
David Pilato
Nicholas Knize
Mikhail Korobov
Lee Hinman
dtpeacock
Janmejay Singh