When an index is deleted we wait on all nodes to ack the delete. Data nodes are expected to both ack the remove of the index from their IndicesService and also the deletion of the store from disk. At the moment all nodes sends this ack which causes wrong counting on the master side. On top of this, we currently have an unneeded WARN message in the logs when client nodes try to acquire locks but do not have a data folder.
Relates to #9605Closes#9672
Groovy was disabled by default, but we turn it on in our test infra. We can then declare support for it so we go and execute script related tests as part of the REST tests suite.
With #9629 we introduced REST spec validation, which barfs whenever the REST spec don't follow the defined conventions. That said, we sometimes execute tests against previous branches and tags which have spec that needs fixing but we can't go back and fix them. We now support the `-Dtests.rest.validate_spec` system property that allows to turn off REST spec validation (enabled by default) so that we can still run tests against old branches/tags.
When multiple fields under object fields share the same name, accessing
by short name is ambiguous. This removes support for short names,
always requiring the full name when used in queries.
closes#8872
Enabling GC logging works now by setting the environment variable ES_GC_LOG_FILE
to the full path to the GC log file. Missing directories will be created as needed.
The ES_USE_GC_LOGGING environment variable is no longer used.
Closes#8471Closes#8479
This moves the rule, so it is made available in the test.jar. In
addition, you can now specify the exception, which triggers a rerun
of the test in order to make it reusable for others.
Also ensured that the NettyTransportTest frees all resources inside
of its testing method instead of pre/post running methods, as those
are still called only once, even though a failed test might be repeated.
At the moment we sometime submit generic runnables, which make life slightly harder when generated pending task list which have to account for them. This commit adds an abstract TimedPrioritizedRunnable class which should always be used. This class also automatically measures time in queue, which is needed for the pending task reporting.
Relates to #8077Closes#9354Closes#9671
Using '_cat/segments' or the indices segments api without matching any index
now returns empty result instead of throwing IndexMissingException.
Closes#9219
Aggregators now return a new collector instance per segment, like Lucene 5 does
with its oal.search.Collector API. This is important for us because things like
knowing whether the field is single or multi-valued is only known at a segment
level.
In order to do that I had to change aggregators to notify their sub aggregators
of new incoming segments (pretty much in the spirit of #6477) while everything
used to be centralized in the AggregationContext class. While this might slow
down a bit deeply nested aggregation trees, this also makes the children
aggregation and the `breadth_first` collection mode much better options since
they can now only replay what they need while they used to have to replay the
whole aggregation tree.
I also took advantage of this big refactoring to remove some abstractions that
were not really required like ValuesSource.MetaData or BucketAnalysisCollector.
I also splitted Aggregator into Aggregator and AggregatorBase in order to
separate the Aggregator API from implementation helpers.
Close#9544
Whenever we have an api that supports GET with a body, we always support the POST method too, as well as providing the body as a query_string parameter called `source`. Our REST spec should reflect this convention. FIxed them and introduced a hard check at parse time in our Java REST tests runner, which will cause the tests to fail if spec are not compliant.
Closes#9629
As a CliTool command could potentially also delete files, the
CheckFileCommand needs to check if those files exist, before
trying to get permissions/owners/groups from that path.
When using the CLI tool infrastructure, a command can potentially write
a new file. In case it overwrites an existing one, you may want to ensure
that the permissions, the owner and the group are kept the same and do not
accidentally change when overwriting those files.
This PR introduces a command that allows you to execute this check per path.
It also adds a new testing dependency, namely jimfs, which allows you to create
in-memory filesystems with certain properties (like supporting or not posix permissions
on this filesystem), so that you can test those features, without executing
tests on a certain operating system.
The FileSystemUtils class has a helper method to create files with
a .new suffix, in case the file, which should be created already
exists. If you install plugins and those have configuration files,
even without changes, you will end up with tons of .new files.
This commit checks the file size and sha-256 sum, and only if those
differ, a .new file is actually being created.
On CI machines node recovery sometimes takes up to 2 seconds. When it happens an update cluster state task gets stuck behind the recovery and tests fail with 1 second timeout. This commit makes sure that we wait for recovery to complete before starting the clock.
This has been very trappy. Rather than continue to allow buggy behavior
of having upgrade/optimize requests sidestep the single shard per node
limits optimize is supposed to be subject to, this removes
the ability to run the upgrade/optimize async.
closes#9638
Unfortunately the lock order is important in the current flush codehe. We have to acquire the readlock fist otherwise
if we are flushing at the end of the recovery while holding the write lock we can deadlock if:
* Thread 1: flushes via API and gets the flush lock but blocks on the readlock since Thread 2 has the writeLock
* Thread 2: flushes at the end of the recovery holding the writeLock and blocks on the flushLock owned by Thread 2
This commit acquires the read lock first which would be done further down anyway for the time of the flush.
As a sideeffect we can now safely flush on calling close() while holding the writeLock.
Today the logic related to deleting an index is spread across several
classes which makes changes to this rather delicate part of the code-base
very difficult. This commit consolidates this logic into the IndicesService
and moves the handling of ack-ing the delete to the master entirely into
`IndicesClusterStateService`.
The percolate api doesn't parse the encoded body provided as `source` query string parameter, when percolating an existing document. Fixed and added REST test that would have caught this since we randomly use GET + encoded `source` param instead of GET + request body in our java runner (the perl runner does the same too).
Closes#9628
_id and _routing now no longer support the 'path' setting on indexes
created with 2.0. Indexes created before 2.0 still support this
setting for backcompat.
closes#6730
Improve cleanup of updateTask timeout handlers. The timeout handlers should be removed as soon as a corresponding update task is processed. Otherwise, timeout handlers might keep old updateTasks and all objects that they are pointing to in memory for the duration of timeout (15 minutes by default).
Fixes#9621
The engine is already pretty complex, it's still confulated with
code that doesn't necessarily belong there. Updateing the settings from
the settings service can be done on the level above. This commit cleans up
the settings code in the engine and moves it to the IndexShard.
"The OpenGIS Abstract Specification: An Object Model for Interoperable Geoprocessing" published by the OGC defines "The boundary of a geometric object is a set of geometric objects of the next lower dimension." The bounding box of a GeometryCollection is therefore the set of bounding rectangles derived from the geometric objects of the next lower dimension. This commit updates the computeBoundingBox and relate methods for the ShapeCollection base class to correctly determine the prefixTree detail level used in Lucene's FilterCellIterator.
closes#9360
Until lately we couldn't close the engine in a tragic event due to
some the lock order and all it's complications. Now that the engine
is much more simplified in terms of having a single IndexWriter etc.
we don't necessarily need the write-lock on close anymore and can
easily just close and continue.
InternalEngine contains a number of inner classes that it uses, however,
this makes the class overly large and hard to extend. In order to be
able to easily add other Engines (such as the ShadowEngine), these
helping methods have been extracted into an AbstractEngine class. The
classes that were previously in `InternalEngine` have been moved to
separate classes, which will allow for better unit testing as well.
None of the functionality of InternalEngine has been changed, this is
only refactoring.
Note that this is a change I originally made on my shadow-replica
branch, however it is easier to review piecemeal so I extracted it into
a separate PR.
Sometimes by the time update settings is called the second node is not in the cluster yet. As a result change of minimum master node settings to 2 is ignored making this test to fail.
Add offset option to 'date_histogram' replacing and simplifying the previous 'pre_offset' and 'post_offset' options.
This change is part of a larger clean up task for `date_histogram` from issue #9062.
Boaz Leskes
Simon Willnauer
javanna
Ryan Ernst
Clinton Gormley
Andreas Kohn
Alexander Reelsen
Victor Buldakov
Christoph Büscher
Adrien Grand
gseng
Igor Motov
Martijn van Groningen
Nicholas Knize
Lee Hinman
Alfredo Serafini