Commit Graph

30230 Commits

Author SHA1 Message Date
Ryan Ernst 3a64e6d121
Test: Remove specifying zip distribution in qa tests (#28805)
Applying the rest test gradle plugin already uses the zip distribution
by default, so specifying it explicitly is not necessary. These are
leftovers from before zip was the default for rest tests.
2018-02-23 13:45:38 -08:00
Ke Li a77273fc01 Reject regex search if regex string is too long (#28542)
* Reject regex search if regex string is too long (#28344)

* Add docs

* Introduce index level setting `index.max_regex_length`
 to control the maximum length of the regular expression

Closes #28344
2018-02-23 10:41:24 -08:00
Luca Cavanna cd3d9c9f80
[TEST] share code between streamable/writeable/xcontent base test classes (#28785)
Today we have two test base classes that have a lot in common when it comes to testing wire and xcontent serialization: `AbstractSerializingTestCase` and `AbstractXContentStreamableTestCase`. There are subtle differences though between the two, in the way they work, what can be overridden and features that they support (e.g. insertion of random fields).

This commit introduces a new base class called `AbstractWireTestCase` which holds all of the serialization test code in common between `Streamable` and `Writeable`. It has two minimal subclasses called `AbstractWireSerializingTestCase` and `AbstractStreamableTestCase` which are specialized for `Writeable` and `Streamable`.

This commit also introduces a new test class called `AbstractXContentTestCase` for all of the xContent testing, which holds a testFromXContent method for parsing and rendering to xContent. This one can be delegated to from the existing `AbstractStreamableXContentTestCase` and `AbstractSerializingTestCase` so that we avoid code duplicate as much as possible and all these base classes offer the same functionalities in the same way. Having this last base class decoupled from the serialization testing may also help with the REST high-level client testing, as there are some classes where it's hard to implement equals/hashcode and this makes it possible to override `assertEqualInstances` for custom equality comparisons (also this base class doesn't require implementing equals/hashcode as it doesn't test such methods.
2018-02-23 10:48:48 +01:00
Nicholas Knize 3728c50d85 [GEO] Fix points_only indexing failure for GeoShapeFieldMapper
This commit fixes a bug that was introduced in PR #27415 for 6.1
and 7.0 where a change to support MULTIPOINT shapes mucked up
indexing of standalone points.
2018-02-22 20:57:09 -06:00
Lee Hinman 5bb79558e7
Decouple XContentGenerator and JsonXContentGenerator from BytesReference (#28772)
This removes the link these two classes have with BytesReference, in favor of an
`InputStream` approach.

Relates to #28504
2018-02-22 14:22:37 -07:00
Ryan Ernst b29ba25c86 Build: Fix packages distributions inclusion of empty directories
This was accidentally broken in #28760.
2018-02-22 11:30:49 -08:00
jxy 497b3d7a20 Fix node ID reported by ThrottlingAllocationDecider (#28779)
Previously the message reported when `node_concurrent_outgoing_recoveries`
resulted in a `THROTTLE` decision included the reporting node's ID rather than
that of the primary. This commit fixes that.

Fixes #28777.
2018-02-22 19:23:15 +00:00
Tim Brooks 5a8ec9b762
Selectors operate on channel contexts (#28468)
This commit is related to #27260. Currently there is a weird
relationship between channel contexts and nio channels. The selectors
use the context for read and writing. But the selector operates directly
on the nio channel for registering, closing, and connecting.

This commit works on improving this relationship. The selector operates
directly on the context which wraps the low level java.nio.channels. The
NioChannel class is simply an API that is used to interact with the
channel (sending messages from outside the selector event loop,
scheduling a close, adding listeners, etc). The context is only used
internally by the channel to implement these apis and by the selector to
perform these operations.
2018-02-22 09:44:52 -07:00
Daniel Mitterdorfer d0831fdd3e
Reduce max javac memory to 512mb (#28783)
With this commit we reduce the maximum amount of memory that the javac
compiler can use from 1g to 512mb. While the build would succeed even
with 256mb, it influences compile time slightly negatively.

We have measured that the runtime overhead stays tolerable by running
the following command five times under repeatable conditions (i.e. we
execute `./gradlew clean`, then drop the caches and TRIM the disk):

```
./gradlew compileGroovy compileJava compileJava9Java\
 compileTestGroovy compileTestJava compileGroovy
```

The results in seconds (as reported by Gradle) are:

* 1gb: avg: 253s, min: 252s, max: 256s
* 512mb: avg: 257s, min: 256s, max: 259s
2018-02-22 16:36:01 +01:00
Tanguy Leroux a6a138905d
Use client settings in repository-gcs (#28575)
Similarly to what has been done for s3 and azure, this commit removes
the repository settings `application_name` and `connect/read_timeout`
in favor of client settings. It introduce a GoogleCloudStorageClientSettings
class (similar to S3ClientSettings) and a bunch of unit tests for that,
it aligns the documentation to be more coherent with the S3 one, it
documents the connect/read timeouts that were not documented at all and
also adds a new client setting that allows to define a custom endpoint.
2018-02-22 15:40:20 +01:00
Jason Tedor daf430c006 Revert "Disable BWC tests for build issues"
This reverts commit 45470945d2.
2018-02-22 06:36:27 -05:00
Luca Cavanna 1df711c5b7
Remove AcknowledgedRestListener in favour of RestToXContentListener (#28724)
This commit makes AcknowledgedResponse implement ToXContentObject, so that the response knows how to print its own content out to XContent, which allows us to remove AcknowledgedRestListener.
2018-02-22 09:13:30 +01:00
Ryan Ernst 014e90d903
Build: Consolidate archives and packages configuration (#28760)
This commit moves the distribution specific tasks into the respective
archives and packages builds. The collocation of common and distribution
specific tasks make it much easier to reason about what is expected in a
particular distribution.
2018-02-21 17:46:40 -08:00
Jason Tedor 3dfb4b8b18 Skip some plugins service tests on Windows
These tests need to be skipped. They cause plugins to be loaded which
causes a child classloader to be opened. We do not want to add the
permissions to be able to close a classloader solely for these tests,
and the JARs created in the test can not be deleted on Windows until the
classloader is closed. Since this will not happen before test teardown,
the test will fail on Windows. So, we skip these tests.
2018-02-21 15:22:27 -05:00
Luca Cavanna 8b4a298874
Migrate some *ResponseTests to AbstractStreamableXContentTestCase (#28749)
This allows us to save a bit of code, but also adds more coverage as it tests serialization which was missing in some of the existing tests. Also it requires implementing equals/hashcode and we get the corresponding tests for them for free from the base test class.
2018-02-21 20:04:12 +01:00
Jason Tedor 45470945d2 Disable BWC tests for build issues
This is a temporary disabling of BWC tests while waiting for some build
issues to be addressed.
2018-02-21 13:25:33 -05:00
Tim Brooks de2a0dfa6e
Ensure that azure stream has socket privileges (#28751)
This is related to #28662. It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
2018-02-21 11:20:06 -07:00
Deb Adair 7715813bc9 [DOCS] Fixed broken link. 2018-02-21 10:04:09 -08:00
Lee Hinman d7eae4b90f
Pass InputStream when creating XContent parser (#28754)
* Pass InputStream when creating XContent parser

Rather than passing the raw `BytesReference` in when creating the xcontent
parser, this passes the StreamInput (which is an InputStream), this allows us to
decouple XContent from BytesReference.

This also removes the use of `commons.Booleans` so it doesn't require more
external commons classes.

Related to #28504

* Undo boolean removal

* Enhance deprecation javadoc
2018-02-21 11:03:25 -07:00
Deb Adair 260d68aad0 [DOCS] Changed to use transient setting to reenabled allocation. Closes #27677 2018-02-21 08:56:06 -08:00
Jason Tedor ed2bbc6e64
Delay path expansion on Windows
There is a bug in the for statement where we execute the JVM options
parser. The bug manfiests in the handling of paths with ) in the
name. The problem is this: we use a for statement to capture the output
of the JVM options parser. A for statement that executes a command
defers execution to cmd. There is this gem from the help:

  1.  If all of the following conditions are met, then quote characters
      on the command line are preserved:

      - no /S switch
      - exactly two quote characters
      - no special characters between the two quote characters,
        where special is one of: &<>()@^|
      - there are one or more whitespace characters between the
        two quote characters
      - the string between the two quote characters is the name
        of an executable file.

  2.  Otherwise, old behavior is to see if the first character is
      a quote character and if so, strip the leading character and
      remove the last quote character on the command line, preserving
      any text after the last quote character.

This means that the ) causes the quotes to be stripped which ruins
everything. This commit fixes this by delaying expansion of the paths.

Relates #28753
2018-02-21 10:58:49 -05:00
javanna e5bceee4a7 [TEST] replace randomAsciiAlphanumOfLengthBetween with randomAsciiLettersOfLengthBetween
This was wrongly changed from the corresponding deprecated method.
2018-02-21 16:36:04 +01:00
Tanguy Leroux 9a95be35cf
[Tests] Extract the testing logic for Google Cloud Storage (#28576)
This pull request extracts in a dedicated class the request/response 
logic that "emulates" a Google Cloud Storage service in our 
repository-gcs tests.

The idea behind this is to make the logic more reusable. The class 
MockHttpTransport has been renamed to MockStorage which now 
only takes care of instantiating a Storage client and does the low-level 
request/response plumbing needed by this client.

The "Google Cloud Storage" logic has been extracted from 
MockHttpTransport and put in a new GoogleCloudStorageTestServer 
that is now independent from the google client testing framework.
2018-02-21 13:20:35 +01:00
Martijn van Groningen 793cbc651a
Moved Grok helper code to a separate Gradle module and let ingest-common module depend on it. 2018-02-21 11:18:08 +01:00
Lokesh Mohanty a878833b25 [Docs] Update links to java9 docs (#28750)
Closes #28683
2018-02-21 10:40:18 +01:00
Yu 7d8fb69d50 version set in ingest pipeline (#27573)
Add support version and version_type in ingest pipelines

Add support for setting document version and version type in set
processor of an ingest pipeline.
2018-02-21 09:34:51 +01:00
Jason Tedor 86e5e38b41 Revert "Add startup logging for standalone tests"
This reverts commit 9e23b7ed72.
2018-02-20 20:00:09 -05:00
Igor Motov c90d0fdf6b Tests: don't wait for completion while trying to get completed task
Nodes are reusing task ids after restart. So in some rare circumstances
the same task id might be assigned to the reindexing task stored by the
old cluster and the new task that is trying to retrieve the task
results. As a result, the get task request can timeout waiting on
itself. Since we already waited for the task to finish before restarting
the cluster, waiting for the task here doesn't make any sense to start
with.

Fixes #28732
2018-02-20 14:14:47 -05:00
Michael Basnight eaa6b41b03 Add 5.6.9 snapshot version 2018-02-20 12:16:02 -06:00
olcbean 50d8a25d67 [Docs] Java high-level REST client : clean up (#28703)
Make doc titles consistent with the ES docs and wrapping 
the code snippets for better readability.
2018-02-20 18:37:53 +01:00
Mikal 5cbafa4eba Updated distribution outputs in contributing docs
This commit updates the path to the output of building distributions
from source in the contributing docs.

Relates #28722
2018-02-20 12:08:11 -05:00
Jim Ferenczi 5991e977d2 Add unreleased v6.2.3 version 2018-02-20 17:49:41 +01:00
Lee Hinman d4fddfa2a0
Remove log4j dependency from elasticsearch-core (#28705)
* Remove log4j dependency from elasticsearch-core

This removes the log4j dependency from our elasticsearch-core project. It was
originally necessary only for our jar classpath checking. It is now replaced by
a `Consumer<String>` so that the es-core dependency doesn't have external
dependencies.

The parts of #28191 which were moved in conjunction (like `ESLoggerFactory` and
`Loggers`) have been moved back where appropriate, since they are not required
in the core jar.

This is tangentially related to #28504

* Add javadocs for `output` parameter

* Change @code to @link
2018-02-20 09:15:54 -07:00
Simon Willnauer b00870600b
Never block on key in `LiveVersionMap#pruneTombstones` (#28736)
Pruning tombstones is best effort and should not block if a key is currently
locked. This can cause a deadlock in rare situations if we switch of append
only optimization while heavily updating the same key in the engine
while the LiveVersionMap is locked. This is very rare since this code
patch only executed every 15 seconds by default since that is the interval
we try to prune the deletes in the version map.

Closes #28714
2018-02-20 16:35:05 +01:00
Luca Cavanna 8bbb3c9ffa
REST high-level client: add support for Rollover Index API (#28698)
Relates to #27205
2018-02-20 15:58:58 +01:00
Jason Tedor 94594f19ab
Fix handling of mandatory meta plugins
This commit fixes an issue with setting plugin.mandatory to include a
meta-plugin. The issue here is that the names that we collect are the
underlying plugins, not the meta-plugin. We should not use the
underlying plugins instead using the names of non-meta plugins and the
names of meta-plugins. This commit addresses this. The strategy here is
that when we look at the installed plugins on the filesystem, we keep
track of which ones are meta-plugins and carry this information up to
where check which plugins are installed against the mandatory plugins.

Relates #28710
2018-02-20 08:57:04 -05:00
Tanguy Leroux 9485b43167 [Tests] Fix RetryHttpInitializerWrapperTests.testIOExceptionRetry
This commit gives more time to the IO exception handler to retry the
request.
2018-02-20 14:54:53 +01:00
Jason Tedor b5d9774c65
Remove leftover doc comment on client JVM check
We previously specified the -server flag to force the JVM to use the
server JVM. This is the default on all the systems that we support when
using a 64-bit JVM (and we no longer support 32-bit JVMs). There was
some trouble with this flag for the Windows service since procrun did
not understand what to do with it; as such, we had to filter this flag
out in the service. When we migrated to parsing JVM options in Java (via
the JVM options parser) we simplified this situation and removed
specifying the -server flag. This commit removes a leftover statement
that we are forcing the server JVM.

Relates #28738
2018-02-20 08:49:26 -05:00
Nhat Nguyen 0c2871c4d2
Replace CAS loop by updateAndGet to improve readability
Relates #28737
2018-02-20 08:03:24 -05:00
Martijn van Groningen e55ce1474d Applied @colings86 changes to the build in order to make new module work in Eclipse too. 2018-02-20 13:49:57 +01:00
Tanguy Leroux 207ca1cc38
[Tests] Simplify GceDiscoverTests (#28726)
GceDiscoverTests can be simplified in a similar manner than #27945. It
now uses a mocked GceInstancesService that exposes internal test cluster
nodes as if they were real GCE nodes. It should also make the test more
robust by not using a HTTP server anymore.

closes #24313
2018-02-20 09:38:22 +01:00
Simon Willnauer 779bc6fd5c
Simplify Engine.Searcher creation (#28728)
Today we have several levels of indirection to acquire an Engine.Searcher.
We first acquire a the reference manager for the scope then acquire an
IndexSearcher and then create a searcher for the engine based on that.
This change simplifies the creation into a single method call instead of
3 different ones.
2018-02-20 09:35:49 +01:00
Simon Willnauer 13a8ba4740 [TEST] Fix flaky IndexServiceTests#testRefreshActuallyWorks 2018-02-20 09:34:20 +01:00
Martijn van Groningen 72de14115b fixed codestyle violation 2018-02-20 08:46:57 +01:00
Martijn van Groningen 9c405e8595 made load method private and add another static getter that users of Grok can use to get the builtin patterns. 2018-02-20 08:09:24 +01:00
Martijn van Groningen 3fad16e76c renamed module 2018-02-20 08:02:02 +01:00
Jason Tedor 105dcb544c
Enable selecting adaptive selection stats
The node stats API enables filtlering the top-level stats for only
desired top-level stats. Yet, this was never enabled for adaptive
replica selection stats. This commit enables this. We also add setting
these stats on the request builder, and fix an inconsistent name in a
setter.

Relates #28721
2018-02-19 16:56:36 -05:00
Jason Tedor 08c5d7def2
Remove leftover mention of file-based scripts
This commit removes a leftover mention of file-based scripts from the
Windows docs.

Relates #28711
2018-02-19 14:52:08 -05:00
Tim Brooks 9215363708
Fix threading issue on listener notification (#28730)
This is a fix for #28729. Currently if a write operation is not properly
queued with a selector we notify the listener. However, we are doing
this by calling a method that is only meant for the selector thread to
call. This commit fixes that issue.
2018-02-19 12:13:00 -07:00
Nhat Nguyen ff2164c4f9
Revisit deletion policy after release the last snapshot (#28627)
We currently revisit the index deletion policy whenever the global
checkpoint has advanced enough. We should also revisit the deletion
policy after releasing the last snapshot of a snapshotting commit. With 
this change, the old index commits will be cleaned up as soon as
possible.

Follow-up of #28140
https://github.com/elastic/elasticsearch/pull/28140#discussion_r162458207
2018-02-19 11:39:15 -05:00