Commit Graph

955 Commits

Author SHA1 Message Date
Christoph Büscher 744b1afcb2 Merge branch 'master' into feature/client_aggs_parsing
Conflicts:
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogramTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsTests.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsTests.java
2017-05-12 10:55:05 +02:00
Simon Willnauer 1155615536 Move DeleteByQuery and Reindex requests into core (#24578)
This allows other plugins to use a client to call the functionality
that is in the core modules without duplicating the logic.
Plugins can now safely send the request and response classes via the
client even if the requests are executed locally. All relevant classes
are loaded by the core classloader such that plugins can share them.

This is re-adds this commit that was revered in 952feb58e4
2017-05-11 20:22:30 +02:00
Simon Willnauer 952feb58e4 Revert "Move DeleteByQuery and Reindex requests into core (#24578)"
This reverts commit 6ea2ae32b8.
2017-05-11 18:26:40 +02:00
Simon Willnauer 6ea2ae32b8 Move DeleteByQuery and Reindex requests into core (#24578)
This allows other plugins to use a client to call the functionality
that is in the core modules without duplicating the logic.
Plugins can now safely send the request and response classes via the
client even if the requests are executed locally. All relevant classes
are loaded by the core classloader such that plugins can share them.
2017-05-11 16:20:40 +02:00
Christoph Büscher 570390ac36 Merge branch 'master' into feature/client_aggs_parsing 2017-05-11 11:20:39 +02:00
Christoph Büscher 64ecf4f4bf Bumping version to v6.0.0-alpha2 2017-05-10 17:12:56 +02:00
Tanguy Leroux bf718a686f Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/terms/InternalTermsTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalExtendedStatsTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalMaxTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalStatsTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/avg/InternalAvgTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/cardinality/InternalCardinalityTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/geobounds/InternalGeoBoundsTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/geocentroid/InternalGeoCentroidTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/min/InternalMinTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/InternalPercentilesTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/hdr/InternalHDRPercentilesRanksTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/InternalTDigestPercentilesRanksTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/sum/InternalSumTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/valuecount/InternalValueCountTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/pipeline/InternalSimpleValueTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/pipeline/bucketmetrics/percentile/InternalPercentilesBucketTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/pipeline/derivative/InternalDerivativeTests.java
#	test/framework/src/main/java/org/elasticsearch/test/InternalAggregationTestCase.java
2017-05-10 13:40:39 +02:00
Ryan Ernst 3f1ef488cd fix checkstyle after script engine rename 2017-05-10 01:04:02 -07:00
Christoph Büscher 1959fd6a1a Merge branch 'master' into feature/client_aggs_parsing 2017-05-09 19:54:36 +02:00
Lee Hinman 1907c46689 Refactor UpdateHelper into unit-testable pieces
This starts breaking up the `UpdateHelper.prepare` method so that each piece can
be individually unit tested. No actual functionality has changed.

Note however, that I did add a TODO about `ctx.op` leniency, which I'd love to
remove as a separate PR if desired.
2017-05-08 09:00:47 -06:00
Jason Tedor 4e5537d9ed Add workaround so path.data can be set in run task
Previously you could set the system property tests.es.path.data and
start the run task with a custom data directory. A change in core to
detect duplicate settings broke this. That change should stay, but we
can work around this easily by only setting path.data to the data
directory if tests.es.path.data is not set.
2017-05-05 15:57:54 -04:00
Tanguy Leroux c7c524dd3f Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-05-05 21:54:37 +02:00
Jason Tedor 035494fa17 Remove obsolete JVM options from build
We start the test JVMs with various options. There are two that we
should remove, they do not make sense.
 - we require Java 8 yet there was a conditional build option for Java 7
 - we do not set MaxDirectMemorySize in our default JVM options, we
   should not in the test JVMs either

Relates #24501
2017-05-05 00:49:18 -04:00
Nik Everett 9bc7e210a0 Test: Move flag to painless tests (#24494)
The `-XX:-OmitStackTraceInFastThrow` flag is only required by Painless's
tests so we'll only set it there. This is much simpler.
2017-05-04 13:11:09 -04:00
James Baiera d928ae210d Add Vagrant based testing fixture (#24249) 2017-05-04 10:17:55 -04:00
Dimitrios Liappis ee6deb34d0 Tests: Switch to fedora-25 in test plugin
Use fedora-25 Vagrant box in VagrantTestPlugin, which was missing from
9a3ab3e800 causing packaging test
failures.

Additionally update TESTING.asciidoc
2017-05-04 14:29:35 +03:00
javanna 62c37339b8 Merge branch 'master' into feature/client_aggs_parsing 2017-05-03 22:04:06 +02:00
Jason Tedor 23801153c7 Fix JVM test argline (#24448)
* Fix JVM test argline

The argline was being overridden by '-XX:-OmitStackTraceInFastThrow'
which led to test failures that were expecting the JVM to be in a
certain state based on the value of tests.jvm.argline but they were not
since these arguments were never passed to the JVM. Additionally, we
need to respect the provided JVM argline if it is already provided with
a flag for OmitStackTraceInFastThrow. This commit fixes this by only
setting OmitStackTraceInFastThrow if it is not already set.

* Add comment

* Fix comment

* More elaborate comment

* Sigh
2017-05-02 19:59:07 -04:00
Simon Willnauer 2f9e9460d4 Move RemoteClusterService into TransportService (#24424)
TransportService and RemoteClusterService are closely coupled already today
and to simplify remote cluster integration down the road it can be a direct
dependency of TransportService. This change moves RemoteClusterService into
TransportService with the goal to make it a hidden implementation detail
of TransportService in followup changes.
2017-05-02 18:09:32 +02:00
Nik Everett 3b47355e56 Try not to lose stacktraces (#24426)
This adds `-XX:-OmitStackTraceInFastThrow` to the JVM arguments
which *should* prevent the JVM from omitting stack traces on
common exception sites. Even though these sites are common, we'd
still like the exceptions to debug them.

This also adds the flag when running tests and adapts some tests
that had workarounds for the absense of the flag.

Closes #24376
2017-05-02 11:34:12 -04:00
javanna 60866da4b7 Merge branch 'master' into feature/client_aggs_parsing 2017-05-02 12:48:30 +02:00
David Pilato 186b401fd8 Upgrade to JUnit 4.12 (#23877)
* Upgrade to JUnit 4.12
* Add permission to junit 4.12 and remove junit4-ant specific permission
2017-05-02 09:58:21 +02:00
Jason Tedor 9525c1c4fa Remove duplicates from checkstyle suppressions
This commit removes a few duplicates from the list of checkstyle
suppressions that accumulated while auto-generating the list of
suppressions based on the files that currently exceed the 140-column
limit.
2017-04-30 12:43:04 -04:00
Jason Tedor e339d894c5 Remove errant entry in checkstyle suppressions
The list of checkstyle suppressions was auto-generated based on the list
of files that currently violate the 140-column limit. An errant entry
was committed that did no harm, but this commit removes it.
2017-04-30 12:15:54 -04:00
Jason Tedor a0f4cde905 Adjust checkstyle suppressions to 140-column limit
We are back to having a 140-column limit. While at some point we will
apply auto-formatting tools to the code base, that is a down-the-road
thing and adjusting the suppressions now shaves minutes off the build
time.

Relates #24408
2017-04-30 12:11:57 -04:00
Christoph Büscher db07a34718 Merge branch 'master' into feature/client_aggs_parsing 2017-04-28 16:42:55 +02:00
Yannick Welsch 35f78d098a Separate publishing from applying cluster states (#24236)
Separates cluster state publishing from applying cluster states:

- ClusterService is split into two classes MasterService and ClusterApplierService. MasterService has the responsibility to calculate cluster state updates for actions that want to change the cluster state (create index, update shard routing table, etc.). ClusterApplierService has the responsibility to apply cluster states that have been successfully published and invokes the cluster state appliers and listeners.
- ClusterApplierService keeps track of the last applied state, but MasterService is stateless and uses the last cluster state that is provided by the discovery module to calculate the next prospective state. The ClusterService class is still kept around, which now just delegates actions to ClusterApplierService and MasterService.
- The discovery implementation is now responsible for managing the last cluster state that is used by the consensus layer and the master service. It also exposes the initial cluster state which is used by the ClusterApplierService. The discovery implementation is also responsible for adding the right cluster-level blocks to the initial state.
- NoneDiscovery has been renamed to TribeDiscovery as it is exclusively used by TribeService. It adds the tribe blocks to the initial state.
- ZenDiscovery is synchronized on state changes to the last cluster state that is used by the consensus layer and the master service, and does not submit cluster state update tasks anymore to make changes to the disco state (except when becoming master).

Control flow for cluster state updates is now as follows:

- State updates are sent to MasterService
- MasterService gets the latest committed cluster state from the discovery implementation and calculates the next cluster state to publish
- MasterService submits the new prospective cluster state to the discovery implementation for publishing
- Discovery implementation publishes cluster states to all nodes and, once the state is committed, asks the ClusterApplierService to apply the newly committed state.
- ClusterApplierService applies state to local node.
2017-04-28 09:34:31 +02:00
Christoph Büscher 1f221154a6 Merge branch 'master' into feature/client_aggs_parsing 2017-04-26 23:09:45 +02:00
Nik Everett fc97e25b56 Add task to look for tests in src/main (#24298)
Creates a new task `namingConventionsMain`, that runs on the
`buildSrc` and `test:framework` projects and fails the build if
any of the classes in the main artifacts are named like tests or
are non-abstract subclasses of ESTestCase.

It also fixes the three tests that would cause it to fail.
2017-04-25 21:11:47 -04:00
Tanguy Leroux 35946a13d0 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/InternalPercentilesTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/percentiles/hdr/InternalHDRPercentilesTests.java
2017-04-20 13:35:21 +02:00
Nik Everett caf376c8af Start building analysis-common module (#23614)
Start moving built in analysis components into the new analysis-common
module. The goal of this project is:
1. Remove core's dependency on lucene-analyzers-common.jar which should
shrink the dependencies for transport client and high level rest client.
2. Prove that analysis plugins can do all the "built in" things by moving all
"built in" behavior to a plugin.
3. Force tests not to depend on any oddball analyzer behavior. If tests
need anything more than the standard analyzer they can use the mock
analyzer provided by Lucene's test infrastructure.
2017-04-19 18:51:34 -04:00
Ryan Ernst e82d8007e3 Build: Add jstack output when starting integ test cluster if timeout occurs (#24193)
This commit adds a call to jstack to see where each node is stuck when
starting up, if a timeout occurs. This also decreases the timeout back
to 30 seconds.
2017-04-19 10:50:47 -07:00
Tanguy Leroux 5717ac3cc6 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing
# Conflicts:
#	core/src/main/java/org/elasticsearch/search/DocValueFormat.java
#	core/src/test/java/org/elasticsearch/search/aggregations/InternalAggregationTestCase.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/InternalMaxTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/avg/InternalAvgTests.java
#	core/src/test/java/org/elasticsearch/search/aggregations/metrics/min/InternalMinTests.java
2017-04-19 10:12:11 +02:00
Tanguy Leroux e81bbc288a Remove Ubuntu 12.04 (#24161)
Ubuntu 12.04 will be EOL on April 28, 2017.
2017-04-19 09:39:42 +02:00
Ryan Ernst 8f666a74f8 Test: Fix plugin integ test to depend on bundling plugin (#24170)
This was broken in the recent refactoring to add dependsOn directly to
cluster configuration.
2017-04-18 17:26:40 -07:00
Adrien Grand 4632661bc7 Upgrade to a Lucene 7 snapshot (#24089)
We want to upgrade to Lucene 7 ahead of time in order to be able to check whether it causes any trouble to Elasticsearch before Lucene 7.0 gets released. From a user perspective, the main benefit of this upgrade is the enhanced support for sparse fields, whose resource consumption is now function of the number of docs that have a value rather than the total number of docs in the index.

Some notes about the change:
 - it includes the deprecation of the `disable_coord` parameter of the `bool` and `common_terms` queries: Lucene has removed support for coord factors
 - it includes the deprecation of the `index.similarity.base` expert setting, since it was only useful to configure coords and query norms, which have both been removed
 - two tests have been marked with `@AwaitsFix` because of #23966, which we intend to address after the merge
2017-04-18 15:17:21 +02:00
Tanguy Leroux 67a9696e55 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-04-18 14:55:58 +02:00
Ryan Ernst a8017ff020 Tests: Move cluster dependencies from runner to cluster (#24142)
After splitting integ tests into cluster configuration and the test
runner task, we still have dependencies of the test runner added as deps
of the cluster. This commit adds dependencies directly to the cluster,
so that the runner can have other dependencies independent of what is
needed for the cluster.
2017-04-17 16:02:46 -07:00
Yannick Welsch 7e581187a7 [TEST] Increase node startup timeouts
Tests are periodically failing on CI as the wait conditions are not met. This commit increases the timeouts.
2017-04-14 10:36:46 +02:00
Ryan Ernst 13489ff3a7 Test: Fix repro line for platformTest (#24095)
This was accidentally being added to packagingTest, which then had two
repro lines.
2017-04-13 14:17:11 -07:00
Tanguy Leroux 7f730c9489 Merge remote-tracking branch 'origin/master' into feature/client_aggs_parsing 2017-04-13 09:24:03 +02:00
Ryan Ernst fb3a281755 Build: Switch jna dependency to an elastic version (#24081)
This new version of jna is rebuilt from the official release of jna, but
with native libs linked against older glibc in order to support all
platforms elasticsearch supports.

closes #23640
2017-04-13 00:17:50 -07:00
Christoph Büscher cf89fb86b5 Merge branch 'master' into feature/client_aggs_parsing 2017-04-12 12:00:40 +02:00
Ryan Ernst 7cebc0fc93 Cleanup outdated comments for fixing up pom dependencies (#24056) 2017-04-11 23:15:30 -07:00
Nik Everett 76603d4413 Move line split in eclipse back to 140 characters
Now that we've moved checkstyle back to 140 we should move the
split.
2017-04-11 13:23:03 -04:00
Simon Willnauer 8cfb9e446c Move back to 140 chars line length (#24013)
This reverts the line limit change in #23623 - this PR doesn't touch the suppression file since we are moving towards automatic code formatting which makes it mainly obsolete.
2017-04-10 17:07:18 +02:00
javanna f538d7b8d6 Merge branch 'master' into feature/client_aggs_parsing 2017-04-10 14:46:57 +02:00
David Roberts 37aadb2adf Add the ability to include extra notices in a plugin's NOTICES file (#23898)
Adds the option for a plugin to specify extra directories containing notices
and licenses files to be incorporated into the overall notices file that is
generated for the plugin.

This can be useful, for example, where the plugin has a non-Java dependency
that itself incorporates many 3rd party components.
2017-04-10 12:37:42 +01:00
Luca Cavanna b73f87b0ea Make buffer limit configurable in HeapBufferedConsumerFactory (#23970)
The buffer limit should have been configurable already, but the factory constructor is package private so it is truly configurable only from the org.elasticsearch.client package. Also the HttpAsyncResponseConsumerFactory interface was package private, so it could only be implemented from the org.elasticsearch.client package.

Closes #23958
2017-04-10 12:27:42 +02:00
Jason Tedor 61c5976aee Upgrade to Log4j 2.8.2
This commit upgrades the Log4j dependencies from version 2.7 to version
2.8.2. This release includes a fix for a case where Log4j could lose
exceptions in the presence of a security manager.

Relates #23995
2017-04-09 07:19:16 -04:00
Ryan Ernst 776006bac5 Collapse repository gcs classes into a single java package (#23975)
This is a single reorge of the classes to simplify making them mostly
package protected.
2017-04-07 11:27:26 -07:00
javanna 67e087088d add class to suppressions list for line length check 2017-04-07 15:08:44 +02:00
Ryan Ernst 203f8433c2 Collapse packages in ec2 discovery plugin (#23909)
This commit collapses all the classes inside ec2 discovery to a single
package name.
2017-04-05 23:51:49 -07:00
Adrien Grand d5d0f140d6 The `filter` and `significant_terms` aggregations should parse the `filter` as a filter, not as a query. (#23797)
This is important for some queries like `bool`, which are parsed differently
depending on whether we want to get a query or a filter.
2017-04-05 16:46:21 +02:00
Ryan Ernst d31d2caf09 Collapse packages in repository-s3 (#23907)
This commit puts all the classes in the repository-s3 plugin into a
single package.  In addition to simplifying the plugin, it will make it
easier to test as things that should be package private will not be
difficult to use inside tests alone.
2017-04-04 15:15:25 -07:00
Jason Tedor 8c554215e0 Ban Boolean#getBoolean
The method Boolean#getBoolean is dangerous. It is too easy to mistakenly
invoke this method thinking that it is parsing a string as a
boolean. However, what it actually does is get a system property with
the specified string, and then attempts to use usual crappy boolean
parsing in the JDK to parse that system property as boolean with
complete leniency (it parses every input value into either true or
false); that is, this method amounts to invoking
Boolean#parseBoolean(String) on the result of
System#getProperty(String). Boo. This commit bans usage of this method.

Relates #23864
2017-04-01 17:02:19 -04:00
Ryan Ernst 1a5c36509f Build: Remove exclusions hack used for gradle 2.13 (#23809)
Now that we are on gradle 3.3, we can take advantage of a fix that was
made in 2.14 which properly handles disabling transitive dependencies in
pom generation.  As it was currently, we actually ended up generated two
exclusions sections in the generated pom. This is yet another example of
why we need validation on the pom files with our generation here, but I
leave that for another day because I still don't know a good way to do
it.
2017-03-29 21:55:48 -07:00
Lee Hinman c8081bde91 Further refactor and extend testing for `TransportShardBulkAction`
This moves `updateReplicaRequest` to `createPrimaryResponse` and separates the
translog updating to be a separate function so that the function purpose is more
easily understood (and testable).

It also separates the logic for `MappingUpdatePerformer` into two functions,
`updateMappingsIfNeeded` and `verifyMappings` so they don't do too much in a
single function. This allows finer-grained error testing for when a mapping
fails to parse or be applied.

Finally, it separates parsing and version validation for
`executeIndexRequestOnReplica` into a separate
method (`prepareIndexOperationOnReplica`) and adds a test for it.

Relates to #23359
2017-03-29 10:56:51 -06:00
Jason Tedor 6b461a1458 Remove line-length violations in SmokeTestClientIT
This commit removes the line-length violations in SmokeTestClientIT.java
and removes this file from the list of suppressions.
2017-03-27 19:59:48 -04:00
Jason Tedor defd0452e7 Modify permissions dialog for plugins
This commit modifies the handling of plugins that require special
permissions to cover a case that was not previously covered.

Relates #23742
2017-03-27 15:52:45 -04:00
Jim Ferenczi 0e95c90e9f Upgrade to Lucene 6.5.0 (#23750) 2017-03-27 15:57:54 +02:00
Ryan Ernst 8a0bc73435 Build: Use targetConfiguration to remove gradle deprecation warning (#23734)
In gradle 3.3 use of getConfiguration on a ModuleDependency was
deprecated. This commit changes it to use getTargetConfiguration.
2017-03-24 09:43:38 -07:00
Jason Tedor 2517cb3062 Fix line-length violations in gce/util/Access
This commit addresses all 100-column line-length violations in
gce/util/Access.java and removes this file from the suppressions list.
2017-03-22 21:34:15 -04:00
Ryan Ernst 25448e3046 Build: remove progress logger hack for gradle 2.13 (#23679)
We are now on minimum gradle 3.3, so we no longer need the groovy/gradle
hacks used to support both 2.13 and 2.14+.
2017-03-21 23:32:40 -07:00
Ryan Ernst ee802ad63c Test: Add dump of integ test cluster logs on failure (#23688)
This commit adds a build listener to the integ test runner which will
print out an excerpt of the logs from the integ test cluster if the test
run fails.  There are future improvements that can be made (for example,
to dump excerpts from dependent clusters like in tribe node tests), but
this is a start, and would help with simple rest tests failures that we
currently don't know what actually happened on the node.
2017-03-21 23:26:07 -07:00
Boaz Leskes 38a18db81e Revert "Revert "Build: Upgrade min gradle to 3.3 (#23544)""
This reverts commit f926a717ed.
2017-03-21 11:20:04 +01:00
Boaz Leskes f926a717ed Revert "Build: Upgrade min gradle to 3.3 (#23544)"
This reverts commit 5d5815416d.
2017-03-21 10:34:07 +01:00
Ryan Ernst 5d5815416d Build: Upgrade min gradle to 3.3 (#23544)
This will allow us to get rid of deprecation warnings that appear when
using 3.3, and also get rid of extra logic for 2.13 required because of
the progress logger.
2017-03-20 21:15:17 -07:00
Jason Tedor 7b17689458 Search took time should use a relative clock
Search took time uses an absolute clock to measure elapsed time, and
then tries to deal with the complexities of using an absolute clock for
this purpose. Instead, we should use a high-precision monotonic relative
clock that is designed exactly for measuring elapsed time. This commit
modifies the search infrastructure to use a relative clock for measuring
took time, but still provides an absolute clock for the components of
search that require a real clock (e.g., index name expression
resolution, etc.).

Relates #23662
2017-03-20 18:48:51 -04:00
Tim Brooks 15a5d1da7e Setup keystore during integration tests (#22966)
This commit creates a keystore and adds settings to it during the
cluster formation for integration tests. Users can define a
`keyStoreSetting` in build files for settings that need to be placed in
the keystore.
2017-03-20 10:56:17 -05:00
Jason Tedor 2eafe8310e Format RemovePluginCommand to 100-column limit
This commit formats RemovePluginCommand.java to the 100-column limit and
removes this file from the list of suppressions.
2017-03-19 22:50:13 -04:00
Jason Tedor 73097f55e7 Format EvilJNANativesTests to 100-column limit
This commit formats EvilJNANativesTests.java to the 100-column limit and
removes this file from the list of suppressions.
2017-03-18 11:13:59 -04:00
Jason Tedor 44d75db9e2 Upgrade from JNA 4.2.2 to JNA 4.4.0
This commit upgrades the JNA dependency from version 4.2.2 to version
4.4.0.

Relates #23636
2017-03-17 21:06:16 -04:00
Nik Everett f30f18285c Eclipse: move print margin to 100 columns
This comes "for free" when you run `gradle eclipse`.
2017-03-17 16:15:44 -04:00
Jason Tedor e8b2f9afd4 Migrate to max line length of 100
This commit moves the checkstyle rule of max line length from 140
characters to 100 characters. We whitelist all existing violations and
will address them in follow-ups.

Relates #23623
2017-03-17 11:19:58 -04:00
Yannick Welsch 5bd14424e6 Fix third-party audit task for Gradle 3.4 (#23612)
In Gradle 3.4, the buildSrc plugin seems to be packaged into a jar before it is accessed by the rest of the build and the signatures file for the third-party audit task cannot be accessed as
getClass().getResource('/forbidden/third-party-audit.txt') then points to a file entry in a JAR, which cannot be loaded directly as a File object. This commit changes the third-party audit task to pass the content of the signatures file as a String instead.
2017-03-17 02:05:46 -07:00
Ryan Ernst 9d4aff524c Build: Split output for jrunscript to stdout and stderr (#23553)
While trying to improve the failure output in #23547, the stderr was
also captured from jrunscript. This was under the assumption that stderr
is only written to in case of an error. However, with java 9, when
JAVA_TOOL_OPTIONS are set, they are output to stderr. And our CI sets
JAVA_TOOL_OPTIONS for some reason. This commit fixes the jrunscript call
to use a separate buffer for stderr.
2017-03-12 08:20:32 -07:00
Jason Tedor c51ef0b2ca Honor max concurrent searches in multi-search
A previous change to the multi-search request execution to avoid stack
overflows regressed on limiting the number of concurrent search requests
from a batched multi-search request. In particular, the replacement of
the tail-recursive call with a loop could asynchronously fire off all of
the remaining search requests in the batch while max concurrent search
requests are already executing. This commit attempts to address this
issue by taking a more careful approach to the initial problem of
recurisve calls. The cause of the initial problem was due to possibility
of individual requests completing on the same thread as invoked the
search action execution. This can happen, for example, in cases when an
individual request does not resolve to any shards. To address this
problem, when an individual request completes we check if it completed
on the same thread as fired off the request. In this case, we loop and
otherwise safely recurse. Sadly, there was a unit test to check that the
maximum number of concurrent search requests was not exceeded, but that
test was broken while modifying the test to reproduce a case that led to
the possibility of stack overflow. As such, we randomize whether or not
search actions execute on the same thread as the thread that invoked the
action.

Relates #23538
2017-03-12 00:45:40 -08:00
Ryan Ernst a2e838d286 Build: Give better output for java version introspection (#23547)
This commit improves the output when jrunscript fails to include the
full output of the command. It also makes the quoting that is needed for
windows only happen on windows (which worked on java 8, but for some
reason does not work with java 9)
2017-03-11 22:12:48 -08:00
Ryan Ernst 57c1d5f821 Build: Fix jrunscript command to quote correctly on windows (#23539)
As noted
in https://github.com/elastic/elasticsearch/issues/22898#issuecomment-277192425
2017-03-11 10:15:51 -08:00
Ryan Ernst 9488985775 Test: Upgrade randomized runner to 2.5.0 (#23513)
This commit upgrades to the newest version of randomized runner. There
is a new additional check that allows ensuring the working directory
for each child jvm is empty. By default, this check will fail the test
run. However, for elasticsearch, we default to wipe the directory. For
example, if you previously told the runner to not wipe the directory, in
order to investigate a failure, the wipe option will delete this data
upon re-running the test.
2017-03-09 11:56:43 -08:00
Ryan Ernst 188c2b3a26 Build: Fix notice generation to use configured notice file (#23472)
While the esplugin extension already had an input for the base notice
file of the plugin, the NoticeTask did not actually know how to use
that, and always used the base notice file from Elasticsearch.
2017-03-02 19:02:43 -08:00
Ryan Ernst 946b9794ad Fix notice generation to use configured notice file 2017-03-02 16:52:09 -08:00
Ryan Ernst 1768efd72d Build: Allow ignoring sha checks for dependencies (#23471)
This commit adds an `ignoreSha` configuration to the `dependencyLicense`
task, which allows to not check for a sha for a given dependency jar.
This is useful for locally built jars, which will constantly change.
2017-03-02 16:49:20 -08:00
Nik Everett a54daade33 Tests InternalSingleBucketAggregation subclasses (#23388)
Adds a common base class for testing subclasses of
`InternalSingleBucketAggregation`. They are so similar they
call into question the utility of having all of these classes.
We maybe could just use `InternalSingleBucketAggregation` in
all those cases.... But for now, let's test the classes!

Relates to #22278
2017-03-01 16:50:52 -05:00
Jason Tedor 640ab1ceab Set build hash to Unknown when unavailable
This commit changes the build hash to be the string "Unknown" when for
some reason this build hash is not available. This aligns the value with
the value we use when the hash is not available from the jar
manifest. This situation can occur when running tests from a worktree
which is not currently handled correctly by JGit, the upstream
dependency that is used to acquire the hash. This causes problems when
running tests locally because the warning header pattern expects a hash
or the string "Unknown". While the warning header pattern be changed to
allow "N/A" as well, it seemed more sensible to align the value here
with the value when the hash is not available from the jar manifest.

Relates #23421
2017-03-01 07:57:14 -05:00
Jim Ferenczi 5c84640126 Upgrade to lucene-6.5.0-snapshot-d00c5ca (#23385)
Lucene upgrade
2017-02-27 18:39:04 +01:00
Ryan Ernst 48280a9403 Tests: Add platformTest to vagrant (#23339)
This change adds a new test task, platformTest, which runs `gradle test
integTest` within a vagrant host. This will allow us to still test on
all the supported platforms, but be able to standardize on the tools
provided in the host system, for example, with a modern version of git
that can allow #22946.

In order to have sufficient memory and cpu to run the tests, the
vagrantfile has been updated to use 8GB and 4 cpus by default. This can
be customized with the `VAGRANT_MEMORY` and `VAGRANT_CPUS` environment
variables.  Also, to save time to show this can work, it currently uses
the same Vagrantfile the packaging tests do. There are a lot of cleanups
we can do to how the gradle-vagrant tasks work, including generating
Vagrantfile altogether, but I think this is fine for now as the same
machines that would run platformTest run packagingTest, and they are
relatively beefy machines, so the higher memory and cpu for them, with
either task, should not be an issue.
2017-02-27 09:21:28 -08:00
Ryan Ernst 37452dbe23 Tests: Fix reproduce line for packagingTest (#23365)
This commit fixes the reproduce line output when the vagrant packagingTest
fails. Before only the `gradle packagingTest` would be output, but the
seed and list of versions was swallowed by groovy with an ancillary
failure (due to the `+` being on the wrong line for a string
continuation).  With the new reproduce line, it is now output next to
the task right after failure, contains the actual task (specific to the
box that fails), and contains the seed. It also no longer contains the
upgrade versions list, as the seed is used to determine which of those
to use, and the same file would be read when testing a failure on a
particular git commit. Finally, this also ties bats test setup directly
to packagingTest, instead of to the vagrant up command.
2017-02-24 19:40:28 -08:00
Ryan Ernst de8049fd2a Tests: Ensure multi node integ tests wait on first node
When a rest integ test has multiple nodes, each node is supposed to not
start configuring itself until the first node has been started, so that
the unicast host information can be written. However, this was never
explicitly setup to occur, and we were just very lucky with the current
gradle version and stability of the code always produced a task graph
that had node0 starting first. With the recent refactorings to integ
tests, the order has changed. This commit fixes the ordering by adding
an explicit dependency between the first node and the other nodes.
2017-02-22 20:54:58 -08:00
Ryan Ernst 74ecd34fd7 Build: Change location in zip of license and notice inclusion for plugins (#23316)
This commit moves the LICENSE.txt and NOTICE.txt files for each plugin
to be alongside the other plugin files, inside the elasticsearch subdir.
This ensures those files are installed alongside the plugin.
2017-02-22 16:13:50 -08:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired (#23304)
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Simon Willnauer ce625ebdcc Expose `batched_reduce_size` via `_search` (#23288)
In #23253 we added an the ability to incrementally reduce search results.
This change exposes the parameter to control the batch since and therefore
the memory consumption of a large search request.
2017-02-21 18:36:59 +01:00
Nik Everett d1de9574ea Checkstyle: Fix link lengths in sampler aggregation 2017-02-17 15:03:57 -05:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
Ryan Ernst fbf824f6a6 Fix plugin properties generation to be exact about the file to copy to
the output zip
2017-02-16 13:43:07 -08:00
Ryan Ernst efb6259e3a Tests: Re-enable debian-8 and fedora-24 packaging tests (#23217)
These images have been rebuilt to be preloaded with java 8 installed.
This change re-enables the systems. It also removes some redundancy in
the rpm checks I found while testing the new images, and fixes a
potential issue with generated resources in plugins where a stale dir
can cause junk to get into the distribution.
2017-02-16 11:26:46 -08:00
Ryan Ernst 6cdf4f3f72 Plugins: Include license and notice files in zip (#23191)
This commit adds the elasticsearch LICENSE.txt to all plugins that
released with elasticsearch, as well as a generated NOTICE.txt specific
to the dependencies of each plugin.
2017-02-15 11:23:12 -08:00
Ryan Ernst 8453cf0622 Build: Add notice file generation (#23170)
This change improves the notice file present in our distributions to
include notice and license files from each included dependency.

closes #22546
2017-02-15 09:40:16 -08:00
Ryan Ernst c4625f230f Build: Support mavenLocal flag in buildSrc (#23157)
When configuring which repositories to pull from, we currently add
mavenLocal() when the `repos.mavenLocal` flag is set. However, this is
only done in normal projects, but not the special buildSrc project. This
change adds that support. Note that this was not possible before gradle
2.13, as there was a bug which prevented sys props from reaching the
buildSrc project (https://issues.gradle.org/browse/GRADLE-2475).
However, we already require 2.13+.
2017-02-14 21:31:56 -08:00
Uwe Schindler 82613198fe Update to forbiddenapis 2.3 (improves Gradle configuration time) (#23154) 2017-02-13 14:30:08 -08:00
Adrien Grand 709cc9ba65 Upgrade to lucene-6.5.0-snapshot-f919485. (#23087) 2017-02-10 15:08:47 +01:00
Tim Brooks fcc568fd8d Add methods requiring connect to forbidden apis (#22964)
This is related to #22116. This commit adds calls that require
SocketPermission connect to forbidden APIs.

The following calls are now forbidden:

- java.net.URL#openStream()
- java.net.URLConnection#connect()
- java.net.URLConnection#getInputStream()
- java.net.Socket#connect(java.net.SocketAddress)
- java.net.Socket#connect(java.net.SocketAddress, int)
- java.nio.channels.SocketChannel#open(java.net.SocketAddress)
- java.nio.channels.SocketChannel#connect(java.net.SocketAddress)
2017-02-07 14:41:50 -06:00
Martijn van Groningen c8d470f190 Change `org.elasticsearch.bootstrap.JNAKernel32Library$SizeT` constructor's modifier to public.
Otherwise `NativeMappedConverter` can't construct this class.

Closes #22991
2017-02-06 15:16:16 +01:00
Adrien Grand c8496fc4f4 Upgrade to Lucene 6.4.1. (#22978) 2017-02-06 09:28:43 +01:00
Ryan Ernst 796087a5a9 Build: Disable fedora for packaging tests (#22974)
Now that debian is disabled, we are seeing similar failures with fedora
not able to install java. This commit temporarily disables fedora until
it is once again stable.
2017-02-03 18:33:53 -08:00
Lee Hinman 39e7c30912 Change certain replica failures not to fail the replica shard
This changes the way that replica failures are handled such that not all
failures will cause the replica shard to be failed or marked as stale.

In some cases such as refresh operations, or global checkpoint syncs, it is
"okay" for the operation to fail without the shard being failed (because no data
is out of sync). In these cases, instead of failing the shard we should simply
fail the operation, and, in the event it is a user-facing operation, return a
5xx response code including the shard-specific failures.

This was accomplished by having two forms of the `Replicas` proxy, one that is
for non-write operations that does not fail the shard, and one that is for write
operations that will fail the shard when an operation fails.

Relates to #10708
2017-02-03 14:39:46 -07:00
Ryan Ernst bbf44394a8 Build: Temporarily remove debian 8 for packaging tests (#22967)
Debian 8 has been having issues with the openjdk package dependencies
being broken. This comment comments out debian-8 from the boxes which
packaging tests will run on CI.
2017-02-03 12:57:21 -08:00
Tim Brooks f70188ac58 Remove connect SocketPermissions from core (#22797)
This is related to #22116. Core no longer needs `SocketPermission`
`connect`.

This permission is relegated to these modules/plugins:
- transport-netty4 module
- reindex module
- repository-url module
- discovery-azure-classic plugin
- discovery-ec2 plugin
- discovery-gce plugin
- repository-azure plugin
- repository-gcs plugin
- repository-hdfs plugin
- repository-s3 plugin

And for tests:
- mocksocket jar
- rest client
- httpcore-nio jar
- httpasyncclient jar
2017-02-03 09:39:56 -06:00
Jason Tedor 9a0b216c36 Upgrade checkstyle to version 7.5
This commit upgrades the checkstyle configuration from version 5.9 to
version 7.5, the latest version as of today. The main enhancement
obtained via this upgrade is better detection of redundant modifiers.

Relates #22960
2017-02-03 09:46:44 -05:00
Ryan Ernst be0236416b Build: Improve detection of older java versions (#22950)
This change switches to using jrunscript, instead of jjs, for detecting
version properties of java, which is available on java versions prior to 8.

closes #22898
2017-02-02 22:58:19 -08:00
Ryan Ernst 7f59bed87b Build: Add nicer error message for pre java 8 (#22911)
This commit adds clearer error message when trying to find the version of java, but
the jjs binary is not found.

closes #22898
2017-02-01 14:07:21 -08:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id (#22206)
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
Jason Tedor 930282e161 Introduce sequence-number-based recovery
This commit introduces sequence-number-based recovery. When a replica
has fallen out of sync, rather than performing a file-based recovery we
first attempt to replay operations since the last local checkpoint on
the replica. To do this, at the start of recovery the replica tells the
primary what its local checkpoint is. The primary will then wait for all
operations between that local checkpoint and the current maximum
sequence number to complete; this is to ensure that there are no gaps in
the operations that will be replayed from the primary to the
replica. This is a best-effort attempt as we currently have no
guarantees on the primary that these operations will be available; if we
are not able to replay all operations in the desired range, we just
fallback to file-based recovery. Later work will strengthen the
guarantees.

Relates #22484
2017-01-27 08:16:38 -08:00
Christoph Büscher 33fd1a606b Remove unneeded checkstyle LineLength check suppression for more classes 2017-01-26 19:40:49 +01:00
Tim Brooks 719e75bb3f Add repository-url module and move URLRepository (#22752)
This is related to #22116. URLRepository requires SocketPermission
connect. This commit introduces a new module called "repository-url"
where URLRepository will reside. With the new module, permissions can
be removed from core.
2017-01-25 17:09:25 -06:00
Nik Everett d704a880e7 Add tests for top_hits aggregation (#22754)
Add unit tests for `TopHitsAggregator` and convert some snippets in
docs for `top_hits` aggregation to `// CONSOLE`.

Relates to #22278
Relates to #18160
2017-01-25 16:15:50 -05:00
Yannick Welsch e37a7d73a2 Remove leftover build files and simplify ProgressLogger usage
These files should have been removed in an earlier commit. This commit also simplifies usage of ProgressLoggerWrapper by using the Groovy delegation instead of using explicit delegation.
2017-01-24 20:44:14 +01:00
Luca Cavanna 47c0e13a3b Stop returning "es." internal exception headers as http response headers (#22703)
move "es." internal headers to separate metadata set in ElasticsearchException and stop returning them as response headers

Closes #17593

* [TEST] remove ESExceptionTests, move its methods to ElasticsearchExceptionTests or ExceptionSerializationTests
2017-01-24 16:12:45 +01:00
Yannick Welsch 12b6ff5233 Use Gradle-version specific source sets
Instead of using Gradle-version specific compilation options, use distinct source sets. This also allows compilation of buildSrc/build-tools under IDEs that
don't understand the version-specific compilation options.

Relates to #22669
2017-01-24 11:09:57 +01:00
Yannick Welsch 36198e0275 Make build Gradle 2.14 / 3.x compatible (#22669)
This changes build files so that building Elasticsearch works with both Gradle 2.13 as well as higher versions of Gradle (tested 2.14 and 3.3), enabling a smooth transition from Gradle 2.13 to 3.x.
2017-01-24 11:09:57 +01:00
Jim Ferenczi 8028578305 Upgrade to Lucene 6.4.0 (#22724)
* Upgrade to Lucene 6.4.0

`ValueSource`s are now converted to `DoubleValueSource`s using the Lucene adapter made for the migration to the new API in 6.4.0.
2017-01-21 04:48:01 +01:00
Jason Tedor 8f6c074691 Revert "Make build Gradle 2.14 / 3.x compatible (#22669)"
This reverts commit 652cb7dbf7.

Relates #22727
2017-01-20 18:16:45 -05:00
Ryan Ernst c5b4bba30b S3 repository: Deprecate specifying credentials through env vars, sys props, and remove profile files (#22567)
* S3 repository: Deprecate specifying credentials through env vars and sys props

This is a follow up to #22479, where storing credentials secure way was
added.
2017-01-19 12:36:32 -08:00
Nik Everett 3ea37ec817 Fix Eclipse project generation
The new plugin causes Eclipse to go a bit crazy. So we skip it
for Eclipse.

Relates to #22646
2017-01-19 13:13:11 -05:00
Tim Brooks a10aa8aade Add TestWithDependenciesPlugin to build (#22646)
This commit adds a MessyRestTestPlugin to the gradle build. It extends 
StandaloneRestTestPlugin. The main piece of functionality that it adds 
is to copy plugin-metadata from dependencies into the 
generated-resources for the current test source. This is necessary to 
ensure that permissions for dependencies are applied when running the 
tests.

A current limitation is that the permissions are applied differently 
than in the distribution sources. When permissions are granted to all 
depedencies for a module or plugin, the permissions are granted to all 
dependencies on the classpath for tests besides a few hardcoded 
exclusions:
- es core
- es test framework
- lucene test framework
- randomized runner
- junit library
2017-01-19 09:43:53 -06:00
Yannick Welsch 652cb7dbf7 Make build Gradle 2.14 / 3.x compatible (#22669)
This changes build files so that building Elasticsearch works with both Gradle 2.13 as well as higher versions of Gradle (tested 2.14 and 3.3), enabling a smooth transition from Gradle 2.13 to 3.x.
2017-01-19 09:56:54 +01:00
Daniel Mitterdorfer aece89d6a1 Make boolean conversion strict (#22200)
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
2017-01-19 07:59:18 +01:00
Nik Everett 1fe74a6b4b Better error when can't auto create index (#22488)
Changes the error message when `action.auto_create_index` or
`index.mapper.dynamic` forbids automatic creation of an index
from `no such index` to one of:
* `no such index and [action.auto_create_index] is [false]`
* `no such index and [index.mapper.dynamic] is [false]`
* `no such index and [action.auto_create_index] contains [-<pattern>] which forbids automatic creation of the index`
* `no such index and [action.auto_create_index] ([all patterns]) doesn't match`

This should make it more clear *why* there is `no such index`.

Closes #22435
2017-01-18 15:18:32 -05:00
Nik Everett e71b26f480 Improve unit test coverage of aggs (#22668)
Add tests for `GlobalAggregator`, `MaxAggregator`, and `InternalMax`.

Relates to #22278
2017-01-18 10:33:45 -05:00
Simon Willnauer f30b1f82ee Remove HttpServer and HttpServerAdapter in favor of a simple dispatch method (#22636)
Today we have quite some abstractions that are essentially providing a simple
dispatch method to the plugins defining a `HttpServerTransport`. This commit
removes `HttpServer` and `HttpServerAdaptor` and introduces a simple `Dispatcher` functional
interface that delegate to `RestController` by default.

Relates to #18482
2017-01-16 21:06:08 +01:00
Alexander Reelsen f6ee6e420b Indexing: Add shard id to indexing operation listener (#22606)
The IndexingOperationListener interface did not provide any
information about the shard id when a document was indexed.

This commit adds the shard id as the first parameter to all methods
in the IndexingOperationListener.
2017-01-16 09:08:16 +01:00
Tanguy Leroux 3a3ce61186 Update Jackson to 2.8.6 (#22596)
closes #22266
2017-01-13 09:05:48 +01:00
Nik Everett 4df2e182bf Fix some typos in error messages in the build 2017-01-10 16:11:05 -05:00
Luca Cavanna ddb93946aa use ElasticsearchException#renderException in BytesRestResponse#convert (#22531) 2017-01-10 20:58:31 +01:00
Nik Everett d50f96e122 Remove InternalAggregation.Type (#22511)
It is no longer needed. It used to contain a lot of strings
used by serialization but those have since been removed. Now
it is just another thing to pass around that we don't really
need.
2017-01-10 11:57:19 -05:00
Tanguy Leroux dc81afd4f8 [TESTS] Create vagrant up task for all boxes (#22496)
Currently, such tasks are only created for default boxes (centos-7, ubuntu-1404) and not all boxes and this can be misleading for developers who want to debug testing scripts on non-default boxes.
2017-01-10 09:45:33 +01:00
javanna 45d4938fcc Migrate some more responses to ToXContentObject 2017-01-06 23:31:48 +01:00
Ryan Ernst 5adb3f992c Merge pull request #22443 from rjernst/disable_assertions
Use tests.asserts flag to allow disabling assertions
2017-01-06 12:03:45 -08:00
Nik Everett 232af512f4 Switch from standalone-test to standalone-rest-test
standalone-rest-test doesn't configure unit tests and for these
integ test only tests, that is what we want.
2017-01-05 10:55:47 +01:00
Nik Everett 812f63e5ef Require either BuildPlugin or StandaloneTestBasePlugin to use RestTestPlugin
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.
2017-01-05 10:55:47 +01:00
javanna f0181b19f5 add REST high level client gradle submodule and first simple method
The RestHighLevelClient class takes as as an argument a low level client instance RestClient. The first method added is ping, which returns true if the call to HEAD / went ok and false if an IOException was thrown. Any other exception gets bubbled up.

There are two kinds of tests, a unit test (RestHighLevelClientTests) that verifies the interaction between high level and low level client, and an integration test (MainActionIT) which relies on an externally started es cluster to send requests to.
2017-01-05 10:55:47 +01:00
Ryan Ernst fff26ba39e Use tests.asserts flag to allow disabling assertions
Randomized runner uses a flag, tests.asserts, which we have previously
not used, but is used in lucene for disabling assertions. This change
modifies the gradle configuration to look for this flag and pass through
to the test runner to determine whether -ea and -esa are added to the
java commandline for tests.
2017-01-04 15:56:53 -08:00
Tim B be22a250b6 Replace Socket, ServerSocket, and HttpServer usages in tests with mocksocket versions (#22287)
This integrates the mocksocket jar with elasticsearch tests. Mocksocket wraps actions requiring SocketPermissions in doPrivilege blocks. This will eventually allow SocketPermissions to be assigned to the mocksocket jar opposed to the entire elasticsearch codebase.
2017-01-04 14:38:51 -06:00
Adrien Grand f8998fece5 Upgrade to lucene-6.4.0-snapshot-084f7a0. (#22413) 2017-01-04 19:03:52 +01:00
javanna 0d67891a64 Remove ParseFieldMatcher usages from QueryParsers#parseRewriteMethod 2017-01-03 15:52:32 +01:00
javanna 41c7d3e092 Remove ParseFieldMatcher usage from Mappers 2017-01-03 15:52:32 +01:00
Jason Tedor 0d40608dc8 Remove leftover checkstyle suppression
This commit removes a leftover checkstyle suppression for a source file
that was temporarily forked into the codebase to hack around a bug in
Log4j. When that source file was removed, the suppression was left
behind.
2017-01-02 14:05:09 -05:00
Jay Modi 1243abfecc build: do not use cached snapshots for backwards compatibility tests
The backwards compatibility tests rely on gradle's built-in mechanisms for resolving dependencies
to get the zip of the older version we test against. By default, this will cache snapshots for
24 hours, which can lead to unexpected failures in CI. This change makes the special configurations
for backwards compatibility always update their snapshots by setting the amount of time to cache
to 0 seconds.
2016-12-23 08:17:16 -05:00
Jason Tedor 432ec54347 Apply logging levels in hierarchical order
This commit adds a test for applying logging levels in hierarchical
order, and addresses an issue with restoring the logging levels at the
end of a test or suite.
2016-12-23 07:51:19 -05:00
Jason Tedor faaa671fb6 Enable assertions in integration tests
When starting a standalone cluster, we do not able assertions. This is
problematic because it means that we miss opportunities to catch
bugs. This commit enables assertions for standalone integration tests,
and fixes a couple bugs that were uncovered by enabling these.

Relates #22334
2016-12-22 20:08:02 -05:00
javanna 55feae1e9a [TEST] remove deleted test classes from checkstyle suppressions
Relates to #21915
2016-12-21 13:28:43 +01:00
Luca Cavanna ae01a51b44 [TEST] make ESSingleNodeTestCase tests repeatable (#22283)
If we conditionally do random things, e.g. initialize a node only after the first test, we have to make sure that we unconditionally create a new seed calling random.nextLong(), then initialize the node under a private randomness context. This makes sure that any random usage through Randomness.get() will retrieve the proper random instance through RandomizedContext.current().getRandom(). When running under private randomness, the context will return the Random instance that was created with the provided seed (forked from the main random instance) rather than the main Random that's exposed to tests as well. Otherwise tests become non repeatable because that initialization part happens only before the first executed test.
2016-12-21 11:44:24 +01:00
Luca Cavanna 3421e54a42 Add fromXContent method to GetResponse (#22082)
Moved field values `toXContent` logic to `GetField` (from `GetResult`), which outputs its own fields, and can also parse them now. Also added `fromXContent` to `GetResult` and `GetResponse`.

 The start object and end object for `GetResponse` output have been moved to `GetResult#toXContent`, from the corresponding rest action. This makes it possible to have `toXContent` and `fromXContent` completely symmetric, as parsing requires looping till an end object is found which is weird when the corresponding `toXContent` doesn't print that out.

This also introduces the foundation for testing retrieval of _source and stored field values.
2016-12-19 17:21:26 +01:00
Boaz Leskes b857b316b6 Add BWC layer to seq no infra and enable BWC tests (#22185)
Sequence BWC logic consists of two elements:

1) Wire level BWC using stream versions.
2) A changed to the global checkpoint maintenance semantics.

For the sequence number infra to work with a mixed version clusters, we have to consider situation where the primary is on an old node and replicas are on new ones (i.e., the replicas will receive operations without seq#) and also the reverse (i.e., the primary sends operations to a replica but the replica can't process the seq# and respond with local checkpoint). An new primary with an old replica is a rare because we do not allow a replica to recover from a new primary. However, it can occur if the old primary failed and a new replica was promoted or during primary relocation where the source primary is treated as a replica until the master starts the target.

1) Old Primary & New Replica - this case is easy as is taken care of by the wire level BWC. All incoming requests will have their seq# set to `UNASSIGNED_SEQ_NO`, which doesn't confuse the local checkpoint logic (keeping it at `NO_OPS_PERFORMED`) 
2) New Primary & Old replica - this one is trickier as the global checkpoint service currently takes all in sync replicas into consideration for the global checkpoint calculation. In order to deal with old replicas, we change the semantics to say all *new node* in sync replicas. That means the replicas on old nodes don't count for the global checkpointing. In this state the seq# infra is not fully operational (you can't search on it, because copies may miss it) but it is maintained on shards that can support it. The old replicas will have to go through a file based recovery at some point and will get the seq# information at that point. There is still an edge case where a new primary fails and an old replica takes over. I'lll discuss this one with @ywelsch as I prefer to avoid it completely.

This PR also re-enables the BWC tests which were disabled. As such it had to fix any BWC issue that had crept in. Most notably an issue with the removal of the `timestamp` field in #21670.

The commit also includes a fix for the default value of the seq number field in replicated write requests (it was 0 but should be -2), that surface some other minor bugs which are fixed as well.

Last - I added some debugging tools like more sane node names and forcing replication request to implement a `toString`
2016-12-19 13:08:24 +01:00
Daniel Mitterdorfer b2aaeb56f3 Update JMH to 1.17.3 2016-12-19 10:02:42 +01:00
Ryan Ernst 9e5cedae23 Fix line lengths in renamed seccomp file 2016-12-16 22:18:56 -08:00
Nik Everett e9bb8d8b38 Don't allow yaml tests with `warnings` that don't skip `warnings` (#21989)
If you write a yaml test with a `warnings` section in a `do` block
that doesn't also have a corresponding `skip` section for `warnings`
then client test runners that don't support `warnings` will fail.
This causes the elasticsearch build to fail so we catch these errors
earlier.

Related to #21811
2016-12-08 13:17:31 -05:00
Nik Everett 0f7c20ae81 Build: NORELEASE is the same as norelease (#22006)
Changes the build to recognize `NORELEASE` as well as `NOCOMMIT` to
mean the same thing as `norelease` and `nocommit` respectively. This
is useful because people have been using them that way but haven't
realized that only the lowercase versions worked.

This also explicitly forbids silly things like `NoReLeAsE` and
`noCOMMIT`, failing the build and telling you to spell them properly.
2016-12-08 11:50:03 -05:00
Simon Willnauer 20177f6eee [TEST] Add back ExternalTestCluster - downstream tests still use it 2016-12-02 10:54:27 +01:00
Jim Ferenczi d791ddf704 Upgrade to lucene-6.4.0-snapshot-ec38570 (#21853)
Set lucene version to 6.4.0-snapshot-ec38570 and update all the sha1s/license
Fix invalid combo after upgrade in query_string query. split_on_whitespace=false is disallowed if auto_generate_phrase_queries=true
Adapt the expectations of some tests to the new format of the Lucene explain output
2016-11-29 18:40:31 +01:00
Yannick Welsch 590a6372ad Disable disk watermarks on REST tests (#21803)
REST tests use the default OOTB low/high disk watermarks of 85%/90%, which can make some tests fail if run on a machine with a fuller disk. This commit changes the watermarks in the same way as in IntegTestCase so that they're essentially ignored.
2016-11-25 19:52:52 +01:00
Luca Cavanna db5a72774b Add indices and filter information to search shards api output (#21738)
Add indices and filter information to search shards api output

The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias.

This is relevant since we have moved parsing of alias filters to the coordinating node.

Relates to #20916
2016-11-22 23:00:25 +01:00
Simon Willnauer a9a2753f0b Add a HostFailureListener to notify client code if a node got disconnected (#21709)
Today there is no way to get notified if a node is disconnected. Client code
must poll the TransportClient constantly to detect that a node is not connected
anymore in order to react and add new nodes or notify altering etc. For instance
if a hostname  gets resolved to an IP but that host is disconnected clients want
to reconnect by resolving the hostname again which is a common situation in cloud
environments.

Closes #21424
2016-11-22 20:46:28 +01:00
Jay Modi 3755c51cd8 Build: add the ability to support plugins in BWC tests
This commit adds the ability to support running with plugins in tests that make use of
backwards compatibility nodes. This can be used to test rolling upgrades with plugins
to ensure they do not cause issues during a rolling upgrade of elasticsearch.
2016-11-22 11:58:17 -05:00
Nik Everett c79371fd5b Remove lang-python and lang-javascript (#20734)
They were deprecated in 5.0. We are concentrating on making
Painless awesome rather than supporting every language possible.

Closes #20698
2016-11-21 22:13:25 -05:00
Tanguy Leroux e7b9e65fc3 Add checkstyle rule to forbid empty javadoc comments (#20881)
This commit adds a RegexpMultiline check to checkstyle that yells when an empty Javadoc comment is found in Java files.

Related #20871
2016-11-21 12:36:44 +01:00
Tanguy Leroux aa871f839a [Tests] Do not pass all env vars to sudo (#21562)
In #21348 the command executed to run the packaging tests has been changed to "sudo -E bats ...", forcing all environment variables from the vagrant user to be passed to the `sudo` command. This breaks a test on opensuse-13 (the one where it checks that elasticsearch cannot be started when `java` is not found) because all the PATH from the user is passed to the sudo command.

This commit restores the previous behavior while allowing only necessary testing environment variables to be passed using a /etc/sudoers.d file.
2016-11-15 15:19:34 +01:00
Yannick Welsch 64a7a960d9 Use pre-JDK9 style FilePermissions on JDK9 (#21540)
JDK9 removed pathname canonicalization when constructing FilePermission objects, which breaks some of the FilePermissions added by Elasticsearch. This commit adds the system property jdk.io.permissionsUseCanonicalPath which makes JDK9 behave like JDK8 w.r.t. FilePermission objects (see #21534).
2016-11-15 09:31:32 +01:00
Jason Tedor 491a945ac8 Add socket permissions for tribe nodes
Today when a node starts, we create dynamic socket permissions based on
the configured HTTP ports and transport ports. If no ports are
configured, we use the default port ranges. When a tribe node starts, a
tribe node creates an internal node client for connecting to each remote
cluster. If neither an explicit HTTP port nor transport ports were
specified, the default port ranges are large enough for the tribe node
and its internal node clients. If an explicit HTTP port or transport
port was specified for the tribe node, then socket permissions for those
ports will be created, but not for the internal node clients. Whether
the internal node clients have explicit ports specified, or attempt to
bind within the default range, socket permissions for these will not
have been created and the internal node clients will hit a permissions
issue when attempting to bind. This commit addresses this issue by also
accounting for tribe nodes when creating the dynamic socket
permissions. Additionally, we add our first real integration test for
tribe nodes.
2016-11-14 11:58:44 -05:00
Yannick Welsch ea65a01789 Use pre-JDK9 style FilePermissions on JDK9
JDK9 removed pathname canonicalization when constructing FilePermission objects, which breaks some of the FilePermissions added by
Elasticsearch. This commit adds the system property jdk.io.permissionsUseCanonicalPath which makes JDK9 behave like JDK8 w.r.t. FilePermissions (see
https://github.com/elastic/elasticsearch/issues/21534).
2016-11-14 14:13:23 +01:00
Tanguy Leroux 7cfa76ec01 Set LoggedExec back 2016-11-14 12:11:53 +01:00
Tanguy Leroux c2c29339d0 Rebasing master 2016-11-14 10:44:06 +01:00
Tanguy Leroux 9cade1cc18 Add Vagrant Gradle plugin
This commit changes the current :elactisearch:qa:vagrant build file and transforms it into a Gradle plugin in order to reuse it in other projects.

Most of the code from the build.gradle file has been moved into the VagrantTestPlugin class. To avoid duplicated VMs when running vagrant tests, the Gradle plugin sets the following environment variables before running vagrant commands:
VAGRANT_CWD: absolute path to the folder that contains the Vagrantfile
VAGRANT_PROJECT_DIR: absolute path to the Gradle project that use the VagrantTestPlugin

The VAGRANT_PROJECT_DIR is used to share project folders and files with the vagrant VM. These folders and files are exported when running the task `gradle vagrantSetUp` which:
- collects all project archives dependencies and copies them into `${project.buildDir}/bats/archives`
- copy all project bats testing files from 'src/test/resources/packaging/tests' into `${project.buildDir}/bats/tests`
- copy all project bats utils files from 'src/test/resources/packaging/utils' into `${project.buildDir}/bats/utils`

It is also possible to inherit and grab the archives/tests/utils files from project dependencies using the plugin configuration:

apply plugin: 'elasticsearch.vagrant'
esvagrant {
    inheritTestUtils true|false
    inheritTestArchives true|false
    inheritTests true|false
}
dependencies {
    // Inherit Bats test utils from :qa:vagrant project
    bats project(path: ':qa:vagrant', configuration: 'bats')
}

The folders `${project.buildDir}/bats/archives`, `${project.buildDir}/bats/tests` and `${project.buildDir}/bats/utils` are then exported to the vagrant VMs and mapped to the BATS_ARCHIVES, BATS_TESTS and BATS_UTILS environnement variables.

The following Gradle tasks have also be renamed:

* gradle vagrantSetUp
This task copies all the necessary files to the project build directory (was `prepareTestRoot`)

* gradle vagrantSmokeTest
This task starts the VMs and echoes a "Hello world" within each VM (was: `smokeTest`)
2016-11-14 10:33:05 +01:00
Adrien Grand 1fd5c47e7f Upgrade to lucene-6.3.0. (#21464) 2016-11-14 09:36:45 +01:00
David Pilato 5336e72fbb Add documentation for Logger with Transport Client
Backport of #21477 in master branch
2016-11-11 21:23:23 +01:00
Jason Tedor 79076334ae Cleanup formatting in UpdateSettingsIT.java
This commit cleans up some code formatting in UpdateSettingsIT.java and
removes this from from the checkstyle line-length supressions.
2016-11-11 12:10:32 -05:00
Lee Hinman 3ae2e9bbc6 Make forbidden APIs be quieter about classpath warnings (#21443)
We currently have a lot of log messages in our CI output like

```
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
[thirdPartyAudit] WARNING: The referenced class 'org.noggit.JSONParser' cannot be loaded. Please fix the classpath!
... repeated 1827281 times ...
```

This changes these messages to be logged at the DEBUG level, so they
will not show up by default.
2016-11-09 14:50:06 -07:00
Ryan Ernst 40a6f46c13 Test: Make ant log for wait condition quieter by default (#21385)
This change simply makes the level of the ant timestamp for waiting on
the integ test cluster echo at the info level instead of warn (the
default) so that it is only output when running with gradle --info, or
when the wait condition fails.
2016-11-08 13:21:57 -08:00
Simon Willnauer ed2865863c Remove LateParsingQuery to prevent timestamp access after context is frozen (#21328)
Today we still have a leftover from older percolators where lucene
query instances where created ahead of time and rewritten later.
This `LateParsingQuery` was resolving `now()` when it's really used which we
don't need anymore. As a side-effect this failed to execute some highlighting
queries when they get rewritten since at that point `now` access it not permitted
anymore to prevent bugs when queries get cached.

Closes #21295
2016-11-04 13:53:03 +01:00
Yannick Welsch d53e1d213f Generate POM files with non-wildcard excludes (#21234)
Dependencies are currently marked as non-transitive in generated POM files by adding a wildcard (*) exclusion. This breaks compatibility with the dependency manager Apache Ivy as it incorrectly translates POMs with * excludes to Ivy XML with * excludes which results in the main artifact being excluded as well (see https://issues.apache.org/jira/browse/IVY-1531). To stay compatible with the current release of Ivy this commit uses explicit excludes for each transitive artifact instead to ensure that the main artifact is not excluded. This should be revisited when we upgrade Gradle to a higher version as the current one (2.13) as Gradle automatically translates non-transitive dependencies to * excludes in 2.14+.
2016-11-04 10:36:38 +01:00
Adrien Grand 2a70f6e7b1 Upgrade to lucene-6.3.0-snapshot-a66a445. (#21309)
This addresses a bug that was introduced with https://issues.apache.org/jira/browse/LUCENE-7501.
2016-11-04 10:34:04 +01:00
Simon Willnauer d77d4fa63a Consume `full_id` request parameter early (#21270)
Since we now validate all consumed request parameter, users can't specify
`_cat/nodes?full_id=true|false` anymore since this parameter is consumed late.
This commit adds a test for this parameter and consumes it before request is processed.

Closes #21266
2016-11-03 10:31:35 +01:00
Simon Willnauer e96fc3aa9f Add a flag to use minimim_master_nodes on the integ test cluster (#21173)
Setting `discovery.initial_state_timeout: 0s` to make `discovery.zen.minimum_master_nodes: N`
work reliably can cause issues in clusters that rely on state recovery once the cluster is available.
This change makes the use or `discovery.zen.minimum_master_nodes` optional for clusters where this behavior is desirable.
2016-10-31 15:27:22 +01:00
Simon Willnauer 326a665509 [TEST] Only set 0s state timeout if we have more than one node 2016-10-28 15:22:02 +02:00
Adrien Grand b3cc54cf0d Upgrade to lucene-6.3.0-snapshot-ed102d6 (#21150)
Lucene 6.3 is expected to be released in the next weeks so it'd be good to give
it some integration testing. I had to upgrade randomized-testing too so that
both Lucene and Elasticsearch are on the same version.
2016-10-28 14:47:15 +02:00
Simon Willnauer 43dbf9c7b6 Use all available hosts in REST tests and allow for real master election (#21161)
Today we only use a single node to send requests to when we run REST tests.
In some cases we have more than one node (ie. in the BWC case) where we should
send requests to all nodes in a round-robin fashion. This change passes all
available node endpoints to the rest test.

Additionally, this change adds the setting of `discovery.zen.minimum_master_nodes`
to the cluster formation forcing the nodes to wait for all other nodes until the cluster
is formed. This allows for a more realistic master election and allows all master eligable
nodes to become master while before always the first node in the cluster became the master.

This also adds logging to each test run to log the master nodes version and the minimum node
version in the cluster to help debugging BWC test failures.
2016-10-28 12:18:47 +02:00
Simon Willnauer 97cc426a89 Fix bwc cluster formation in order to run BWC tests against a mixed version cluster (#21145)
This fixes our cluster formation task to run REST tests against a mixed version cluster.
Yet, due to some limitations in our test framework `indices.rollover` tests are currently
disabled for the BWC case since they select the current master as the merge node which
happens to be a BWC node and we can't relocate all shards to it since the primaries are on
a higher version node. This will be fixed in a followup.

Closes #21142

Note: This has been cherry-picked from 5.0 and fixes several rest tests
as well as a BWC break in `OsStats.java`
2016-10-27 17:03:53 +02:00
Christoph Büscher aaf3477327 Remove checkstyle LineLenght check suppression from 23 more classes 2016-10-26 15:24:21 +02:00
Simon Willnauer 0a410d3916 Pass executor name to request interceptor to support async intercept calls (#21089)
Today the request interceptor can't support async calls since the response
of the async call would execute on a different thread ie. a client or listener
thread. This means in-turn that the intercepted handler is not executed with the
thread it was supposed to run and therefor can, if it's executing blocking
operations, potentially deadlock an entire server.
2016-10-24 13:57:07 +02:00
Ryan Ernst 53cff0f00f Move all zen discovery classes into o.e.discovery.zen (#21032)
* Move all zen discovery classes into o.e.discovery.zen

This collapses sub packages of zen into zen. These all had just a couple
classes each, and there is really no reason to have the subpackages.

* fix checkstyle
2016-10-20 00:44:48 -07:00
Ryan Ernst dca614aa3b Build: Change `gradle run` to use zip distribution (#21001)
When running `gradle run`, a developer usually intends to get a running
instance as if they had run elasticsearch from the command line. This is
different than the isolated environment we use for integration testing
plugins. This change switches the run task to use the zip distribution,
so that all modules included in the normal distribution are included.
2016-10-18 11:48:58 -07:00
Christoph Büscher 274d5d1fdb Remove org.junit.rules.ExpectedException from forbidden-apis again 2016-10-18 11:13:09 +02:00
Christoph Büscher abff485d68 Use Lucenes expectThrows() when testing exceptions
Cleaning up a few remaining occurences of using junits ExpectedException rule in
favor of using LuceneTestCase#expectThrows() which is more concise and versatile.
2016-10-18 10:34:48 +02:00
Tanguy Leroux 3b578db365 MultiGet should not fail entirely if alias resolves to many indices (#20858)
MultiGet should not fail entirely when one of the items of a multi get request refers to an alias that points to multiple indices.

closes #20845
2016-10-11 18:07:39 +02:00
Simon Willnauer c98e3f60f7 Ensure source filtering automatons are only compiled once (#20857)
This change adds a overloaded `XContentMapValues#filter` method that returns
a function enclosing the compiled automatons that can be reused across filter
calls. This for instance prevents compiling automatons over and over again when
hits are filtered or in the SourceFieldMapper for each document.

Closes #20839
2016-10-11 15:30:38 +02:00
Boaz Leskes fc4645bae6 Cluster Settings Updates should not trigger circuit breakers. (#20827)
Settings updates are important to be able to help and administer a cluster in distress. We shouldn't block it due to circuit breakers. An extreme example is where we are actually trying to increase and unreasonable low setting for the circuit breaker itself.

See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/242/
2016-10-11 10:36:55 +02:00
Lee Hinman bd8f546381 Enable incremental compilation in gradle (#20847) 2016-10-10 15:47:35 -06:00
Nik Everett 6418f89feb Remove NodeServicesProvider
Instead provide services where they are needed. The class worked
well as a temporary measure to easy removal of guice from the index
level but now we can remove it entirely.

-1 @Inject annotation
2016-10-08 07:35:58 -04:00
Jason Tedor 535bf92280 Upgrade Log4j 2 to version 2.7
This commit upgrades the Log4j 2 dependency to version 2.7 and removes
some hacks that we had in place to work around bugs in Log4j 2 version
2.6.2.

Relates #20805
2016-10-07 13:05:30 -05:00
Ryan Ernst 6da0f0dcc0 Build: Fix plugin poms to have correct artifact id (#20764)
We already override the name in plugin pom files to be that configured
for the plugin but we also need to explicitly set the artifactId.
2016-10-05 08:10:27 -07:00
Daniel Mitterdorfer aa308fe159 Upgrade microbenchmarks to JMH 1.15 2016-10-04 10:20:45 +02:00
Ryan Ernst a82bf7a851 Build: Use more general archivesBaseName setting, to ensure pom uses
matching file name
2016-10-03 21:40:43 -07:00
Nik Everett 7cfc2f126c CONSOLEify the first half of getting-started
Adds a `// TESTRESPONSE[_cat]` syntax that sets up some substitutions
useful for asserting that _cat APIs match.
2016-10-01 12:56:23 +02:00
Simon Willnauer fe1803c957 Remove AnalysisService and reduce it to a simple name to analyzer mapping (#20627)
Today we hold on to all possible tokenizers, tokenfilters etc. when we create
an index service on a node. This was mainly done to allow the `_analyze` API to
directly access all these primitive. We fixed this in #19827 and can now get rid of
the AnalysisService entirely and replace it with a simple map like class. This
ensures we don't create a gazillion long living objects that are entirely useless since
they are never used in most of the indices. Also those objects might consume a considerable
amount of memory since they might load stopwords or synonyms etc.

Closes #19828
2016-09-23 08:53:50 +02:00
Jason Tedor 78ceb9301b Fix cluster wait condition
Today we define a cluster wait condition to try to wait at least a
certain number of nodes when running integration tests. Alas, the wait
condition is incorrect because wait_for_nodes>=${numNodes} will be split
by parameter parsing on the equals sign so the request looks like it has
a parameter named wait_for_nodes>. The fact that REST param parsing is
lenient leads to this being undiscovered. This commit fixes this issue.

Relates #20601
2016-09-21 16:33:15 -04:00
Ali Beyad 5031824291 File-based discovery plugin integration tests (#20492)
Adds an integration test for the file-based discovery plugin
to test the plugin operates correctly and uses the hosts
configured in `unicast_hosts.txt` with a real cluster

Closes #20459
2016-09-21 15:48:18 -04:00
Nik Everett 6705c6aa2f Further document the pom hack
We have a "HUGE HACK" that allows us to publish zip artifacts to
Sonatype's OSS repository without javadoc and source jars. We don't
include those jars because the zip is just a repackaging of the
core and module jars for which we already publish the javadoc and
source jars. So we have a hack to publish the zip artifact when the
pom says the project is of type 'pom'.
2016-09-21 11:25:19 -04:00
Yannick Welsch 47088cc206 Allow build to run even if there is no Git remote named origin (#20608)
The build currently depends on the presence of a Git remote named origin to determine the URL that is used in the generated POM file. As this is best-effort anyhow and only required by Maven Central, this commit allows the build to run even if a Git remote with the name "origin" is missing.
2016-09-21 17:22:06 +02:00
Yannick Welsch 191fadafcc Fix logger usage checker for Log4j 2
With the switch to Log4j 2 throughout our code base, the logger usage checker was temporarily disabled. This commit
adapts the checks to work with Log4j 2 and re-enables the Gradle checks.

Closes #20243
2016-09-21 14:44:14 +02:00
Nik Everett af285f1aa4 Convert more docs to CONSOLE
* reference/api-conventions.asciidoc
* reference/redirects.asciidoc

Relates to #18160
2016-09-20 13:12:12 -04:00
Nik Everett 3771d80830 Fix up nebulaRealPom
I'm not sure why we need this pom instead of the pom generated by
nebula, but if we are going to have it then we need to populate it
with appropriate stuff like project name, description, and url.
2016-09-20 11:05:23 -04:00
Ryan Ernst 6921b4a66b Build: Add back hack for client copying client jar pom (#20577)
Gradle appears to have a bug in maven publshing which will not match the
artifactId of a generated pom with the artifact id it puts in the file.
This adds back a copy hack from the original pom file name to the client
pom file name (which we had before #20403 inadvertently
removed it).
2016-09-20 01:37:18 -07:00
Tanguy Leroux 38dce6384f Improve TribeIT tests
This commit adds a new test TribeIT#testClusterStateNodes() to verify that the tribe node correctly reflects the nodes of the remote clusters it is connected to.

It also changes the existing tests so that they really use two remote clusters now.
2016-09-20 10:31:32 +02:00
Nik Everett 9fd42d1d48 Fix allowed non-CONSOLE docs on windows
`\` vs `/` strikes again. This time I just convert all the `\` into
`/`. It isn't perfect but anyone who sees it'll figure out what is
up.
2016-09-19 22:43:17 -04:00
Ryan Ernst 85b8f29415 Build: Remove old maven deploy support (#20403)
* Build: Remove old maven deploy support

This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.

Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.

relates #20164
closes #20375

* Remove unnecessary extra spacing
2016-09-19 15:10:41 -07:00
Ryan Ernst e2854bb3a4 Build: Remove maven central snapshots from repositories (#20567)
We were using maven snapshots during heavy development, but this should
not be something generally available (we should never release depending
on a snapshot version in maven). This change removes the snapshot repo.
If we ever need it temporarily for some reason, we can add it if/when
it is necessary.

relates #20559
2016-09-19 15:06:05 -07:00
Nik Everett 156393be0e Fail build if new doc snippets aren't `// CONSOLE`
This tracks the snippets that probably should be converted to
`// CONSOLE` or `// TESTRESPONSE` and fails the build if the list
of files with such snippets doesn't match the list in `docs/build.gradle`.
Setting the file looks like
```
/* List of files that have snippets that probably should be converted to
 * `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and
 * only remove entries from this list. When it is empty we'll remove it
 * entirely and have a party! There will be cake and everything.... */
buildRestTests.expectedUnconvertedCandidates = [
  'plugins/discovery-azure-classic.asciidoc',
...
  'reference/search/suggesters/completion-suggest.asciidoc',
]
```

This list is in `build.gradle` because we expect it to be fairly
temporary. In a few months we'll have converted all of the docs and won't
ned it any more.

From now on if you add now docs that contain a snippet that shows an
interaction with elasticsearch you have three choices:
1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the
responses. The build (specifically (`gradle docs:check`) will test that
these interactions "work". If there isn't a `// TESTRESPONSE` snippet
then "work" just means "Elasticsearch responds with a 200-level response
code and no `WARNING` headers. This is way better than nothing.

2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with
Elasticsearch. This should only be required for stuff like javascript
source code or `curl` against an external service like AWS or GCE. The
snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be
tested.

3. Add `// TEST[skip:reason]` under the snippet. This will just skip the
snippet in the test phase. This should really be reserved for snippets
where we can't test them because they require an external service that
we don't have at testing time.

Please, please, please, please don't add more things to the list. After
all, it sais there'll be cake when we remove it entirely!

Relates to #18160
2016-09-19 16:43:43 -04:00
Ali Beyad 50584c4103 Merge pull request #20532 from rjernst/rolling_upgrades
This PR introduces backward compatibility index tests to test the rolling upgrade process amongst Elasticsearch instances within the same major version. The test executes in three phases. In the first phase, we form a cluster of 2 ES instances on an old version. In the second phase, we keep one of the nodes from the old cluster, kill the other node, but preserve its data directory and start an instance of the current version of ES using the same data directory as the killed instance. In the third phase, we kill the other old version ES instance from the first phase and launch a new instance, using the same data directory as the killed instance. Therefore, during phase 3, we have fully migrated and have all current versions of ES running. In each phase, we run REST tests that index documents and search them, ensuring at each stage that the documents from the previous phase are still there.

Note that because we haven't released a GA yet of 5.0, the tests currently don't start an old version cluster in the first phase. Once GA is released, this will be changed to make the backward compatibility version 5.0, while the current version in the cluster will be 5.x.
2016-09-19 16:14:38 -04:00
David Pilato a530399ae5 Merge branch 'pr/remove-mapper-attachments' 2016-09-19 14:47:14 +02:00
Daniel Mitterdorfer 6f628557b9 Upgrade from JMH 1.12 to JMH 1.14 2016-09-19 14:28:16 +02:00
David Pilato dfd1eebdd0 Remove mapper attachments plugin
We now have in 5.0.0 `ingest-attachment` plugin.
We can remove `mapper-attachments` plugin for 6.0.

Closes #18837.
2016-09-19 09:01:16 +02:00
Ali Beyad 98230d035a Adds a preserveIndicesUponCompletion method to ESRestTestCase
that can be overridden by subclasses if the test must not
delete indices it created after exiting.
2016-09-16 19:21:26 -04:00
Ali Beyad 83adc87015 Removes stopNodeUponCompletion in favor of moving the stop
nodes task to the final part of the cluster test task execution
graph.
2016-09-16 16:32:43 -04:00
Ali Beyad ce86ed1fdd Merge remote-tracking branch 'upstream/master' into rolling_upgrades 2016-09-16 10:43:38 -04:00
Ali Beyad 56f97500c6 In the rolling upgrades tests, we do not want to stop nodes
automatically between tasks, as we want some of the nodes from
the previous task to continue running in the next task. This
commit enables a cluster configuration setting to not stop
nodes automatically after a task runs, but instead the creator
of the test task must stop the running nodes explicitly in a
cleanup phase.
2016-09-16 10:36:55 -04:00
Ali Beyad ba072ec18e When Elasticsearch nodes are started in gradle to form a
cluster, we wait for the cluster health to indicate the
necessary nodes have formed a cluster.  This check was an
exact value (equality) check.  However, if we are trying to
connect the nodes in the cluster to nodes from a previously
formed cluster (of the same name), then we will have more
nodes returned by the cluster health check than the current
task's configured number of nodes.  Hence, this check needs
to be a >= check.  This commit fixes it.
2016-09-16 09:41:26 -04:00
Boaz Leskes 4ac3b416f2 Don't generate html outputs from checkstyle (#20481)
We run our builds with `gradle --info` and that ends up generating horrific long dumps of xslt.
2016-09-14 17:23:40 +02:00
Ali Beyad 0c7dd1865c Reworking yaml tests for rolling upgrades 2016-09-12 18:46:06 -04:00
chico chen 4cf4683a64 Build: can not find JAVA_HOME in eclipse gradle plugin. (#17645) 2016-09-12 18:38:24 -04:00
Jason Tedor 3b90906f4f Adapt release flag to new CLI arguments
The JDK project is in the process of modifying the command-line flags
for various JDK tools (http://openjdk.java.net/jeps/293). In particular,
the release flag on javac has changed from -release to --release. This
commit adapts the build process to this change.

Relates #20420
2016-09-12 10:53:32 -04:00
javanna 90ab460fcc move parsing of search ext sections to the coordinating node 2016-09-09 19:10:42 +02:00
javanna dc2ba90f48 clarify that SearchParseElement is only used for custom fetch sub phases and clean up extension point
SearchParseElement is renamed to FetchSubPhaseParser and moved to the search.fetch package. Its parse method doesn't get the SearchContext as argument anymore, only the XContentParser, and the return type is what gets parsed (the fetch sub phase context which we may as well rename later).

It is the parser that initializes the FetchSubPhaseContext then. SearchService retrieves the parser by name, calls parse against it and stores the result of parsing by name. No need for FetchSubPhase.ContextFactory anymore, which can be removed.
2016-09-09 18:05:49 +02:00
javanna 4b57a0fd97 resolve some line length problems and remove some entry from checkstyle suppressions (deleted classes) 2016-09-09 18:05:49 +02:00
Simon Willnauer f319545814 Prepare master branch to be 6.0.0-alpha1 2016-09-08 12:55:30 +02:00
Ryan Ernst bd2de367cc Use 5.0 alpha5 for bwc version, so we have transport.ports
Also fixed bug to ensure unicast host is writtent in yaml quotes
2016-09-06 16:08:27 -07:00
Ryan Ernst e6ac27fcfa Merge branch 'master' into rolling_upgrades 2016-09-06 15:52:03 -07:00
Ryan Ernst a844b085f1 Made node config always have a unicast transport uri closure 2016-09-06 15:51:14 -07:00
Jason Tedor 0d7dfcd798 Merge pull request #20338 from jasontedor/remove-plugin
Print message when removing plugin with config
2016-09-06 11:43:51 -04:00
Nik Everett 5cff2a046d Remove most of the need for `// NOTCONSOLE`
and be much more stingy about what we consider a console candidate.

* Add `// CONSOLE` to check-running
* Fix version in some snippets
* Mark groovy snippets as groovy
* Fix versions in plugins
* Fix language marker errors
* Fix language parsing in snippets

  This adds support for snippets who's language is written like
  `[source, txt]` and `["source","js",subs="attributes,callouts"]`.

  This also makes language required for snippets which is nice because
  then we can be sure we can grep for snippets in a particular language.
2016-09-06 10:32:54 -04:00
Jason Tedor e081b2b2e8 Remove length violation in RemovePluginCommand
This commit removes a line-length violation in RemovePluginCommand.java
and removes this file from the list of files for which the line-length
check is suppressed.
2016-09-06 07:28:05 -04:00
Ali Beyad 5d8aa6b4fe Adds tests for rolling upgrades to execute 2016-09-03 01:34:39 -04:00
Jason Tedor b9966fed36 Hack around Log4j bug rendering exceptions
Log4j has a bug where it does not handle a security exception that can
be thrown when it is rendering a stack trace. This commit intentionally
introduces jar hell with the ThrowableProxy class to work around this
bug until a fix is a released.

Relates #20306
2016-09-02 20:26:32 -04:00
javanna e5a741ab67 fix line length in some touched classes 2016-09-02 10:23:49 +02:00
Martijn van Groningen a110498ad8 settings: Make `action.auto_create_index` setting a dynamic cluster setting.
Closes #7513
2016-09-01 12:33:30 +02:00
Simon Willnauer a0becd26b1 Optimize indexing for the autogenerated ID append-only case (#20211)
If elasticsearch controls the ID values as well as the documents
version we can optimize the code that adds / appends the documents
to the index. Essentially we an skip the version lookup for all
documents unless the same document is delivered more than once.

On the lucene level we can simply call IndexWriter#addDocument instead
of #updateDocument but on the Engine level we need to ensure that we deoptimize
the case once we see the same document more than once.

This is done as follows:

1. Mark every request with a timestamp. This is done once on the first node that
receives a request and is fixed for this request. This can be even the
machine local time (see why later). The important part is that retry
requests will have the same value as the original one.

2. In the engine we make sure we keep the highest seen time stamp of "retry" requests.
This is updated while the retry request has its doc id lock. Call this `maxUnsafeAutoIdTimestamp`

3. When the engine runs an "optimized" request comes, it compares it's timestamp with the
current `maxUnsafeAutoIdTimestamp` (but doesn't update it). If the the request
timestamp is higher it is safe to execute it as optimized (no retry request with the same
timestamp has been run before). If not we fall back to "non-optimzed" mode and run the request as a retry one
and update the `maxUnsafeAutoIdTimestamp` unless it's been updated already to a higher value

Relates to #19813
2016-09-01 10:39:40 +02:00
Ryan Ernst ecaf6ef001 Add rolling upgrade test project 2016-08-31 16:45:03 -07:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Jason Tedor 4e69ac0272 Add link to open logger usage issue
This commit adds comments linking to an open issue regarding updating
the logger usage check for the Log4j 2 API.
2016-08-30 21:13:17 -04:00
Ryan Ernst e19f2b6348 Tests: Improve rest suite names and generated test names for docs tests
Rest test suites are currently only the directory above the yaml test
file. That is confusing when there are more than one directory level
which contain yaml tests, as there are in generated docs tests. This
change makes rest tests use the full relative path to the rest test root
as the suite name, and also makes the test names for docs tests a little
clearer (that they are testing an example from a specific line number,
instead of just the line number as an opaque test name).
2016-08-30 13:55:44 -07:00