Percolator queries and index alias filters are parsed once and reused as long as they exist on a node. If they contain time based range filters with a `now` expression then the alias filters and percolator queries are going to be incorrect from the moment these are constructed (depending on the date rounding).
If a range filter or range query is constructed as part of adding a percolator query or a index alias filter then these get wrapped in special query or filter wrappers that defer the resolution of now at last possible moment as apposed during parse time. In the case of the range filter a special Resolvable Filter makes sure that `now` is resolved when the DocIdSet is pulled and in the case of the range query `now` is resolved at query rewrite time. Both occur at the time the range filter or query is used as apposed when the query or filter is constructed during parse time.
Closes#8474Closes#8534
Today recovery sources are not cancled if a shard is closed. The recovery target
is already cancled when shards are closed but we should also cleanup and cancel
the sources side since it holds on to shard locks / references until it's closed.
Today we hold on to search context reference if they are not cleaned
up for a while until a reaper thread trashes them if they timed out.
This commit removes all pending contexts once the index is closed to release
resources and filehandles immediatly once the index is closed.
During node shutdown we have a race condition between processing cluster state updates (creating shards) and closing down the index service. This may cause shards to leak and not be closed properly.
This commit removes the concurrency in shard closing as InternalIndexService.removeShard has been synchronized for a long time now.
On the other hand, the commit restores the parallel shutdown of indices lost in 7e1d8a6ca3Closes#8557
was set to yes, dangling indices were deleted by mistake,
because a RemoveDanglingIndices runnable was added
to every dangling indices, without considering the auto_import_dangled
setting.
PR #8464 come with a bug in the example provided.
First, the current log file is not compressed so it should not end with `.gz`.
Second, conversion pattern was removing all the log content but was printing only the log date.
Then, the current log filename was hardcoded to `elasticsearch` instead of the cluster name.
Make also LogConfigurator#ALLOWED_SUFFIXES package private so that it can be used in LoggingConfigurationTests, now that it's in the same package as the class that it tests.
Add few randomized aspects to LoggingConfigurationTests.
Make sure that files such as logging.yml.rpmnew or logging.yml.bak are not loaded as logging configuration.
Only files that start with the "logging." prefix and end with ".yaml", ".yml", ".json" and ".properties" suffix get loaded.
Closes#7457
Interrupting a thread while blocking on an NIO Read / Write Operation
can cause a file to be closed due to the interrupts. This can have unpredictable
effects when files are open by index readers etc. we should prevent interruptions
across the board if possible.
Closes#8494
We need to register those data paths otherwise we might miss path that
need to get cleaned when using local gatway etc. which can otherwise
cause imports of dangeling indices.
This commit fixes the issue caused by restore process deleting all legacy checksum files at the end of restore process. Instead it keeps the latest version of the checksum intact. The issue manifests itself in losing checksum for all legacy files restored into post 1.3.0 cluster, which in turn causes unnecessary snapshotting of files that didn't change.
Fixes#8119
Now each error is reported in bulk response rather than causing entire bulk to fail.
Added a Junit test but the use of TransportClient means the error is manifested differently to a REST based request - instead of a NullPointer the whole of the bulk request failed with a RoutingMissingException. Changed TransportBulkAction to catch this exception and treat it the same as the existing logic for a ElasticsearchParseException - the individual bulk request items are flagged and reported individually rather than failing the whole bulk request.
Closes#8365
Test used `indices.recovery.concurrent_streams` when creating an index but this is a node setting. Moved it to the node settings and added similar settings to speed up concurrent recoveries.
Also fixed a misleading log message in ShardRecoveryHandler when logging a remove corruption
Today it's possible that the data directory for a single shard is used by more than on
IndexShard->Store instances. While one shard is already closed but has a concurrent recovery
running and a new shard is creating it's engine files can conflict and data can potentially
be lost. We also remove shards data without checking if there are still users of the files
or if files are still open which can cause pending writes / flushes or the delete operation
to fail. If the latter is the case the index might be treated as a dangeling index and is brought
back to life at a later point in time.
This commit introduces a shard level lock that prevents modifications to the shard data
while it's still in use. Locks are created per shard and maintined in NodeEnvironment.java.
In contrast to most java concurrency primitives those locks are not reentrant.
This commit also adds infrastructure that checks if all shard locks are released after tests.
Don't eagerly cache parent type filters in bitset cache or nested object fields that are leafs.
Also let parent/child queries not rely on FixedBitSetFilter, but rather on regular Filter
Closes#8440
Fixed behaviour where two representations of the default index analyzer weren't being treated as equivalent. Added REST test to confirm fix.
Closes#2716
In addition to `_source`, the following variables are available through
the `ctx` map: `_index`, `_type`, `_id`, `_version`, `_routing`,
`_parent`, `_timestamp`, `_ttl`.
Some of these fields are more useful still within the context of an
Update By Query, see #1607, #2230, #2231.
While this commit is primariy a fix for issue/8433 it adds more rigor to ShapeBuilder for parsing against the GeoJSON specification. Specifically, this adds LinearRing and LineString validity checks as defined in http://geojson.org/geojson-spec.html to ensure valid polygons are specified. The benefit of this fix is to provide a gate check at parse time to avoid any further processing if an invalid GeoJSON is provided. More parse checks like these will be necessary going forward to ensure full compliance with the GeoJSON specification.
Closes#8433
Based on some test failures, this commit fixes two minor things
* Bind ports only on so called ephemeral ports to prevent try to
bind to ports where elasticsearch already runs on
* Remove @Network annotation as it was used in a wrong scope
Martijn van Groningen
Simon Willnauer
Boaz Leskes
Philipp Bogensberger
David Pilato
Nicholas Knize
javanna
Mathias Fussenegger
Igor Motov
Lee Hinman
Michael McCandless
markharwood
Ken Wu
Andreas Kohn
Adrien Grand
Ryan Doyle
nwarz
Olivier Favre
Alex Ksikes
Alexander Reelsen
Colin Goodheart-Smithe