* Adds task dependenciesInfo to BuildPlugin to generate a CSV file with dependencies information (name,version,url,license)
* Adds `ConcatFilesTask.groovy` to concatenates multiple files into one
* Adds task `:distribution:generateDependenciesReport` to concatenate `dependencies.csv` files into a single file (`es-dependencies.csv` by default)
# Examples:
$ gradle dependenciesInfo :distribution:generateDependenciesReport
## Use `csv` system property to customize the output file path
$ gradle dependenciesInfo :distribution:generateDependenciesReport -Dcsv=/tmp/elasticsearch-dependencies.csv
## When branch is not master, use `build.branch` system property to generate correct licenses URLs
$ gradle dependenciesInfo :distribution:generateDependenciesReport -Dbuild.branch=6.x -Dcsv=/tmp/elasticsearch-dependencies.csv
This change fixes a bug when a keyword term in the `after` key is not present in the shard.
In this case the global ord of the document values are compared with the insertion point of the
`after` keyword and values that are equal to the insertion point should be considered "after" the top value.
We disable these tests because the JVM flags we use will not start with
JDK 10. Since we do not support running these old versions on newer JDKs
anyway, this commit disables testing of these when running on JDK 10.
In PR #27965, we set the global checkpoint from the translog in a store
recovery. However, we set after an engine is opened. This causes the
global checkpoint assertion in TranslogWriter violated as if we are
forced to close the engine before we set the global checkpoint. A
closing engine will close translog which in turn read the current global
checkpoint; however it is still unassigned and smaller than the initial
global checkpoint from translog.
Closes#27970
Today we always recover a primary from the last commit point. However
with a new deletion policy, we keep multiple commit points in the
existing store, thus we have chance to find a good starting commit
point. With a good starting commit point, we may be able to throw away
stale operations. This PR rollbacks a primary to a starting commit then
recovering from translog.
Relates #10708
Non-Peer recoveries should restore the global checkpoint rather than wait for the activation of the primary. This brings us a step closer to a universe where a recovered shard always has a valid global checkpoint. Concretely:
1) Recovery from store can read the checkpoint from the translog
2) Recovery from local shards and snapshots can set the global checkpoint to the local checkpoint as this is the only copy of the shard.
3) Recovery of an empty shard can set it to `NO_OPS_PERFORMED`
Peer recoveries will follow but require more work and thus will have their own PR.
I also used the moment to clean up `IndexShard`'s api around starting the engine and doing recovery from the translog. The current naming are a relic of the past and don't align with the current naming schemes in the engine.
This assertion does not hold if a shard is recovered from an empty store
but failed then retries. Moreover, if the openMode is CREATE_INDEX_*, we
pass CREATE mode to the IndexWriterConfig to create a new index and
overwrite the existing one.
Closes#27960
We disabled the Javadoc task on JDK 10 due to an apparent bug in Javadoc
generation on JDK 10. However, the client JAR task sets up its own
Javadoc task for client JARs (because it merely copies the non-client
Javadoc JAR). This commit diables that task too, since the Javadocs for
the non-client JAR will not exist.
Relates #27962
When a new snapshot is created it is added to the cluster state as a
snapshot-in-progress in INIT state, and the initialization is kicked
off in a new runnable task by SnapshotService.beginSnapshot(). The
initialization writes multiple files before updating the cluster state
to change the snapshot-in-progress to STARTED state. This leaves a
short window in which the snapshot could be deleted (let's say, because
the snapshot is stuck in INIT or because it takes too much time to
upload all the initialization files for all snapshotted indices). If
the INIT snapshot is deleted, the snapshot-in-progress becomes ABORTED
but once the initialization in SnapshotService.beginSnapshot() finished
it is change back to STARTED state again.
This commit avoids an ABORTED snapshot to be started if it has been
deleted during initialization. It also adds a test that would have failed
with the previous behavior, and changes few method names here and there.
This commit changes some Azure tests so that they do not rely on
MockZenPing and TestZenDiscovery anymore, but instead use a mocked
AzureComputeService that exposes internal test cluster nodes as if
they were real Azure nodes.
Related to #27859Closes#27917, #11533
There appears to be a bug in JDK 10 for generating Javadocs with some
nested anonymous classes. This commit disables these on JDK 10 until the
upstream issue is resolved.
Relates #27952
Today we don't persist the global checkpoint when finishing a peer
recovery even though we advance an in memory value. This commit persists
the global checkpoint in RecoveryTarget#finalizeRecovery.
Closes#27861
Search response listeners should not be exposed in search plugin.
Support was added but reverted right after (not present in any release).
Though the SearchPlugin still contains a default definition for search response listeners
due to a broken revert. This change removes this extension point that is basically no-op.
In the case of nested aggregator this allows it to push down buffered child docs down to children aggregator.
Before this was done as part of the `NestedAggregator#getLeafCollector(...)`, but by then the children aggregators
have already moved on to the next segment and this causes incorrect results to be produced.
Closes#27912
* Limit the analyzed text for highlighting
- Introduce index level settings to control the max number of character
to be analyzed for highlighting
- Throw an error if analysis is required on a larger text
Closes#27517
* Splits nio project into two for eclipse build only
https://github.com/elastic/elasticsearch/pull/27801 introduced a new gradle project `:libs:elasticsearch-nio` which creates cyclical project dependencies when importingthe projects into Eclipse.
This change applies the same trick as we have for the core project where, and building for Eclipse, splits the `:libs:elasticsearch-nio` project into `:libs:elasticsearch-nio` which points to `src/main` and `:libs:elasticsearch-nio-tests` which points to `src/test`. This prevents cyclical project dependencies in Eclipse arising from the fact that eclipse does not separate compile/runtime dependencies from test dependencies.
* Removes integTest bits since there are none
Lucene TopDocs collector are now able to early terminate the collection
based on the index sort. This change plugs this new functionality directly in the
query phase instead of relying on a dedicated early terminating collector.
These templates were generated with 5.0. We need those generated with 6.0 since
we do not guarantee compatibility with previous versions of the template anyway.
I removed the winlogbeat template which is a bit harder to generate as it requires
Windows, since we do not aim to be exhaustive.
This is related to #27802. This commit adds a jar called
elasticsearch-nio that contains the base nio classes that will be used
for the tcp nio transport and eventually the http nio transport.
The jar does not depend on elasticsearch:core, so all references to core
have been removed.
JDK 10 has gone fully-modular. This means that:
- when targeting JDK 8 with a JDK 10 compiler, we have to use the full
profile
- when targeting JDK 10 with a JDK 10 compiler, we have to use
-add-modules java.base
Today we only target JDK 8 (our minimum compatibility version) so we
need to change the compiler flags conditional on using a JDK 10
compiler. This commit does that.
Relates #27884
Today when we get a metadata snapshot directly from a store directory,
we acquire a metadata lock, then acquire an IndexWriter lock. However,
we create a CheckIndex in IndexShard without acquiring the metadata lock
first. This causes a recovery failed because the IndexWriter lock can be
still held by method snapshotStoreMetadata. This commit makes sure to
create a CheckIndex under the metadata lock.
Closes#24481Closes#27731
Relates #24787
Previously to this change when DocStats are added together (for example when adding the index size of all primary shards for an index) we naively added the `totalSizeInBytes` together. This worked most of the time but not when the index size on one or multiple shards was reported to be `-1` (no value).
This change improves the logic by considering if the current value or the value to be added is `-1`:
* If the current and new value are both `-1` the value remains at `-1`
* If the current value is `-1` and the new value is not `-1`, current value is changed to be equal to the new value
* If the current value is not `-1` and the new value is `-1` the new value is ignored and the current value is not changed
* If both the current and new values are not `-1` the current value is changed to be equal to the sum of the current and new values.
The change also re-enables the failing rollover YAML test that was failing due to this bug.
TestZenDiscovery is used to allow discovery based on in memory structures. This isn't a relevant for the cloud providers tests (but isn't a problem at the moment either)
We used to shrink the version map under an external lock. This is
quite ambigious and instead we can simply issue an empty refresh to
shrink it.
Closes#27852
Today we prevent that the same thread acquires the same lock more than once.
This restriction is a relict form the early days of this concurrency construct
and can be removed.
The last operation executed in IndicesClientDocumentationIT.testCreate()
is an asynchronous index creation. Because nothing waits for its
completion, on slow machines the index can sometimes be created after
the testCreate() test is finished, and it can fail the following test.
Closes#27754
While the LiveVersionMap is an internal class that belongs to the engine we do
rely on some external locking to enforce the desired semantics. Yet, in tests
we mimic the outer locking but we don't have any way to enforce or assert on
that the lock is actually hold. This change moves the KeyedLock inside the
LiveVersionMap that allows the engine to access it as before but enables
assertions in the LiveVersionMap to ensure the lock for the modifying or
reading key is actually hold.
Currently FiltersAggregationBuilder#doRewrite creates a new FiltersAggregationBuilder which doesn't correctly copy the original "keyed" field if a non-keyed filter gets rewritten.
This can cause rendering bugs of the output aggregations like the one reported in #27841.
Closes#27841
Doug Turnbull
Maxime Gréau
Jim Ferenczi
Tanguy Leroux
Jason Tedor
Nhat Nguyen
Boaz Leskes
Ryan Ernst
Alexander Reelsen
olcbean
Martijn van Groningen
Mayya Sharipova
Colin Goodheart-Smithe
Jim Ferenczi
Adrien Grand
Andy Bristol
Tim Brooks
Sian Lerk Lau
Simon Willnauer
Stuart Cam
Christoph Büscher