We currently have a lot of log messages in our CI output like
```
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
... repeated 1827281 times ...
```
This changes these messages to be logged at the DEBUG level, so they
will not show up by default.
This change simply makes the level of the ant timestamp for waiting on
the integ test cluster echo at the info level instead of warn (the
default) so that it is only output when running with gradle --info, or
when the wait condition fails.
Today we still have a leftover from older percolators where lucene
query instances where created ahead of time and rewritten later.
This `LateParsingQuery` was resolving `now()` when it's really used which we
don't need anymore. As a side-effect this failed to execute some highlighting
queries when they get rewritten since at that point `now` access it not permitted
anymore to prevent bugs when queries get cached.
Closes#21295
Dependencies are currently marked as non-transitive in generated POM files by adding a wildcard (*) exclusion. This breaks compatibility with the dependency manager Apache Ivy as it incorrectly translates POMs with * excludes to Ivy XML with * excludes which results in the main artifact being excluded as well (see https://issues.apache.org/jira/browse/IVY-1531). To stay compatible with the current release of Ivy this commit uses explicit excludes for each transitive artifact instead to ensure that the main artifact is not excluded. This should be revisited when we upgrade Gradle to a higher version as the current one (2.13) as Gradle automatically translates non-transitive dependencies to * excludes in 2.14+.
Since we now validate all consumed request parameter, users can't specify
`_cat/nodes?full_id=true|false` anymore since this parameter is consumed late.
This commit adds a test for this parameter and consumes it before request is processed.
Closes#21266
Setting `discovery.initial_state_timeout: 0s` to make `discovery.zen.minimum_master_nodes: N`
work reliably can cause issues in clusters that rely on state recovery once the cluster is available.
This change makes the use or `discovery.zen.minimum_master_nodes` optional for clusters where this behavior is desirable.
Lucene 6.3 is expected to be released in the next weeks so it'd be good to give
it some integration testing. I had to upgrade randomized-testing too so that
both Lucene and Elasticsearch are on the same version.
Today we only use a single node to send requests to when we run REST tests.
In some cases we have more than one node (ie. in the BWC case) where we should
send requests to all nodes in a round-robin fashion. This change passes all
available node endpoints to the rest test.
Additionally, this change adds the setting of `discovery.zen.minimum_master_nodes`
to the cluster formation forcing the nodes to wait for all other nodes until the cluster
is formed. This allows for a more realistic master election and allows all master eligable
nodes to become master while before always the first node in the cluster became the master.
This also adds logging to each test run to log the master nodes version and the minimum node
version in the cluster to help debugging BWC test failures.
This fixes our cluster formation task to run REST tests against a mixed version cluster.
Yet, due to some limitations in our test framework `indices.rollover` tests are currently
disabled for the BWC case since they select the current master as the merge node which
happens to be a BWC node and we can't relocate all shards to it since the primaries are on
a higher version node. This will be fixed in a followup.
Closes#21142
Note: This has been cherry-picked from 5.0 and fixes several rest tests
as well as a BWC break in `OsStats.java`
Today the request interceptor can't support async calls since the response
of the async call would execute on a different thread ie. a client or listener
thread. This means in-turn that the intercepted handler is not executed with the
thread it was supposed to run and therefor can, if it's executing blocking
operations, potentially deadlock an entire server.
* Move all zen discovery classes into o.e.discovery.zen
This collapses sub packages of zen into zen. These all had just a couple
classes each, and there is really no reason to have the subpackages.
* fix checkstyle
When running `gradle run`, a developer usually intends to get a running
instance as if they had run elasticsearch from the command line. This is
different than the isolated environment we use for integration testing
plugins. This change switches the run task to use the zip distribution,
so that all modules included in the normal distribution are included.
Cleaning up a few remaining occurences of using junits ExpectedException rule in
favor of using LuceneTestCase#expectThrows() which is more concise and versatile.
This change adds a overloaded `XContentMapValues#filter` method that returns
a function enclosing the compiled automatons that can be reused across filter
calls. This for instance prevents compiling automatons over and over again when
hits are filtered or in the SourceFieldMapper for each document.
Closes#20839
Settings updates are important to be able to help and administer a cluster in distress. We shouldn't block it due to circuit breakers. An extreme example is where we are actually trying to increase and unreasonable low setting for the circuit breaker itself.
See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/242/
Instead provide services where they are needed. The class worked
well as a temporary measure to easy removal of guice from the index
level but now we can remove it entirely.
-1 @Inject annotation
This commit upgrades the Log4j 2 dependency to version 2.7 and removes
some hacks that we had in place to work around bugs in Log4j 2 version
2.6.2.
Relates #20805
Today we hold on to all possible tokenizers, tokenfilters etc. when we create
an index service on a node. This was mainly done to allow the `_analyze` API to
directly access all these primitive. We fixed this in #19827 and can now get rid of
the AnalysisService entirely and replace it with a simple map like class. This
ensures we don't create a gazillion long living objects that are entirely useless since
they are never used in most of the indices. Also those objects might consume a considerable
amount of memory since they might load stopwords or synonyms etc.
Closes#19828
Today we define a cluster wait condition to try to wait at least a
certain number of nodes when running integration tests. Alas, the wait
condition is incorrect because wait_for_nodes>=${numNodes} will be split
by parameter parsing on the equals sign so the request looks like it has
a parameter named wait_for_nodes>. The fact that REST param parsing is
lenient leads to this being undiscovered. This commit fixes this issue.
Relates #20601
Adds an integration test for the file-based discovery plugin
to test the plugin operates correctly and uses the hosts
configured in `unicast_hosts.txt` with a real cluster
Closes#20459
We have a "HUGE HACK" that allows us to publish zip artifacts to
Sonatype's OSS repository without javadoc and source jars. We don't
include those jars because the zip is just a repackaging of the
core and module jars for which we already publish the javadoc and
source jars. So we have a hack to publish the zip artifact when the
pom says the project is of type 'pom'.
The build currently depends on the presence of a Git remote named origin to determine the URL that is used in the generated POM file. As this is best-effort anyhow and only required by Maven Central, this commit allows the build to run even if a Git remote with the name "origin" is missing.
With the switch to Log4j 2 throughout our code base, the logger usage checker was temporarily disabled. This commit
adapts the checks to work with Log4j 2 and re-enables the Gradle checks.
Closes#20243
I'm not sure why we need this pom instead of the pom generated by
nebula, but if we are going to have it then we need to populate it
with appropriate stuff like project name, description, and url.
Gradle appears to have a bug in maven publshing which will not match the
artifactId of a generated pom with the artifact id it puts in the file.
This adds back a copy hack from the original pom file name to the client
pom file name (which we had before #20403 inadvertently
removed it).
This commit adds a new test TribeIT#testClusterStateNodes() to verify that the tribe node correctly reflects the nodes of the remote clusters it is connected to.
It also changes the existing tests so that they really use two remote clusters now.
* Build: Remove old maven deploy support
This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.
Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.
relates #20164closes#20375
* Remove unnecessary extra spacing
We were using maven snapshots during heavy development, but this should
not be something generally available (we should never release depending
on a snapshot version in maven). This change removes the snapshot repo.
If we ever need it temporarily for some reason, we can add it if/when
it is necessary.
relates #20559
This tracks the snippets that probably should be converted to
`// CONSOLE` or `// TESTRESPONSE` and fails the build if the list
of files with such snippets doesn't match the list in `docs/build.gradle`.
Setting the file looks like
```
/* List of files that have snippets that probably should be converted to
* `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and
* only remove entries from this list. When it is empty we'll remove it
* entirely and have a party! There will be cake and everything.... */
buildRestTests.expectedUnconvertedCandidates = [
'plugins/discovery-azure-classic.asciidoc',
...
'reference/search/suggesters/completion-suggest.asciidoc',
]
```
This list is in `build.gradle` because we expect it to be fairly
temporary. In a few months we'll have converted all of the docs and won't
ned it any more.
From now on if you add now docs that contain a snippet that shows an
interaction with elasticsearch you have three choices:
1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the
responses. The build (specifically (`gradle docs:check`) will test that
these interactions "work". If there isn't a `// TESTRESPONSE` snippet
then "work" just means "Elasticsearch responds with a 200-level response
code and no `WARNING` headers. This is way better than nothing.
2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with
Elasticsearch. This should only be required for stuff like javascript
source code or `curl` against an external service like AWS or GCE. The
snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be
tested.
3. Add `// TEST[skip:reason]` under the snippet. This will just skip the
snippet in the test phase. This should really be reserved for snippets
where we can't test them because they require an external service that
we don't have at testing time.
Please, please, please, please don't add more things to the list. After
all, it sais there'll be cake when we remove it entirely!
Relates to #18160
This PR introduces backward compatibility index tests to test the rolling upgrade process amongst Elasticsearch instances within the same major version. The test executes in three phases. In the first phase, we form a cluster of 2 ES instances on an old version. In the second phase, we keep one of the nodes from the old cluster, kill the other node, but preserve its data directory and start an instance of the current version of ES using the same data directory as the killed instance. In the third phase, we kill the other old version ES instance from the first phase and launch a new instance, using the same data directory as the killed instance. Therefore, during phase 3, we have fully migrated and have all current versions of ES running. In each phase, we run REST tests that index documents and search them, ensuring at each stage that the documents from the previous phase are still there.
Note that because we haven't released a GA yet of 5.0, the tests currently don't start an old version cluster in the first phase. Once GA is released, this will be changed to make the backward compatibility version 5.0, while the current version in the cluster will be 5.x.
Adrien Grand
David Pilato
Jason Tedor
Lee Hinman
Ryan Ernst
Simon Willnauer
Yannick Welsch
Simon Willnauer
Christoph Büscher
Tanguy Leroux
Boaz Leskes
Nik Everett
Daniel Mitterdorfer
Ali Beyad