Commit Graph

25029 Commits

Author SHA1 Message Date
Luca Cavanna 293a3cab01 Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries (#21378)
* Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries

Turns out that AbstractAsyncResponseConsumer from apache async http client is stateful and cannot be reused across multiple requests. The failover mechanism was mistakenly reusing that same instance, which can be provided by users, across retries in case nodes are down or return 5xx errors. The downside is that we have to change the signature of two public methods, as HttpAsyncResponseConsumer cannot be provided directly anymore, rather its factory needs to be provided which is going to be used to create one instance of the consumer per request attempt.

Up until now we tested our RestClient against multiple nodes only in a mock environment, where we don't really send http requests. In that scenario we can verify that retries etc. work properly but the interaction with the http client library in a real scenario is different and can catch other problems. With this commit we also add an integration test that sends requests to multiple hosts, and some of them may also get stopped meanwhile. The specific test for pathPrefix was also removed as pathPrefix is now randomly applied by default, hence implicitly tested. Moved also a small test method that checked the validity of the path argument to the unit test RestClientSingleHostTests.

Also increase default buffer limit to 100MB and make it required in default consumer

The default buffer limit used to be 10MB but that proved not to be high enough for scroll requests (see reindex from remote). With this commit we increase the limit to 100MB and make it a bit more visibile in the consumer factory.
2016-11-08 16:42:42 +01:00
Jason Tedor 68a94e711a Remove redundant Javadocs from G1GC check
This commit removes some unnecessary Javadocs from the G1GC bootstrap
check. The code here is self-explanatory.
2016-11-08 10:32:10 -05:00
Jason Tedor a1ef6e9635 Cleanup Javadocs in BootstrapCheck.java
This commit cleans up the formatting of some Javadocs in
BootstrapCheck.java, and corrects some docs that had become stale as the
bootstrap checks evolved.
2016-11-08 10:16:06 -05:00
Nik Everett 31be683e57 Skip alias name validation rest test on 5.0 cluster
Alias name validation was introduced in 5.1.
2016-11-08 09:36:47 -05:00
Yannick Welsch 14a0e8ee57 Remove mutable status field from cluster state (#21379)
The ClusterState class currently has a mutable volatile field "status" that is only used by the ClusterStateObserver to differentiate between a cluster state that is being applied or one that has already been applied. This commit removes the field from cluster state, making it a truly immutable class. This information is stored instead by ClusterService, which is the only place that should update this field (PublishClusterStateAction was also updating it, but that information was never used anywhere). A new class is introduced (ClusterServiceState) which emcompasses the current cluster state as well as the current status, which is only used by the ClusterStateObserver mechanism.
2016-11-08 14:37:09 +01:00
Jason Tedor 6f6c633298 Fix markup in sysconfig-file docs
This commit fixes a markup issue in the sysconfig-file docs, adding a
missing backtick.
2016-11-08 08:33:45 -05:00
Nik Everett 3787ea27bf Validate alias names the same as index names
Applied (almost) the same rules we use to validate index names
to new alias names. The only rule not applies it "must be lowercase".
We have tests that don't follow that rule and I except there are lots
of examples of camelCase alias names in the wild. We can add that
validation but I'm not sure it is worth it.

Closes #20748

Adds an alias that starts with `#` to the BWC index and validates
that you can read from it and remove it. Starting with `#` isn't
allowed after 5.1.0/6.0.0 so we don't create the alias or check it
after those versions.
2016-11-08 08:23:12 -05:00
Jason Tedor cc2743743c Fix version constant in Groovy API docs
This commit fixes the version constant in the Groovy API docs from
5.0.0-alpha5 to 6.0.0-alpha1.
2016-11-08 08:08:45 -05:00
Joeyn414 aefffca763 minor grammatical fix (#21384) 2016-11-08 11:52:52 +01:00
Colin Goodheart-Smithe 6614cc73cc Fixes compile error in Eclipse in ClusterRerouteRequestTests (#21400)
Before this change Eclipse (4.6.1) would show compile errors in `ClusterRerouteRequestTests` for the elements in `RANDOM_COMMAND_GENERATORS`. This seems to be because the eclipse compiler does not recognised that the elements in the list are all `Supplier`s of bjects that are subclasses of `AllocationCommand`.

This change fixes the problem by adding a generics hint to the `Arrays.toList()` call.
2016-11-08 10:26:55 +00:00
Yannick Welsch cd34eed03e Make ensureGreen and ensureYellow wait for cluster size consistency (#21344)
We currently often use ensureGreen or ensureYellow to check whether the cluster is in a good state again after shutting down a node. With the change in #21092, however, it can happen that if the node that is stopped is the master node, another node will become master and publish a cluster state where it is master but where the node that was stopped hasn't been removed yet from the cluster state. It will only publish a second state thereafter where the old master is removed. If the ensureGreen/ensureYellow is timed just right, it will get to execute before the second cluster state update removing the old master and the condition ensureGreen / ensureYellow might not hold at that point anymore.
2016-11-08 11:07:54 +01:00
Ryan Ernst 6b4280c7be Better fix for java8 restriction of g1gc check
This makes the fix actually testable.
2016-11-07 17:04:41 -08:00
Ryan Ernst 562a30d3c6 Move licenses for core jar to core directory (#21383)
All plugins currently have their own licenses dir for the
dependencyLicenses task, but core disables this and has the check inside
distribution. This may have been better for maven, but for
gradle it makes more sense to just use the dependencyLicenses task that
automatically exists inside :core, and remove the hacked up version that
is inside distribution.
2016-11-07 15:29:35 -08:00
Ryan Ernst 7a2c984bcc Test: Remove multi process support from rest test runner (#21391)
At one point in the past when moving out the rest tests from core to
their own subproject, we had multiple test classes which evenly split up
the tests to run. However, we simplified this and went back to a single
test runner to have better reproduceability in tests. This change
removes the remnants of that multiplexing support.
2016-11-07 15:07:34 -08:00
Ryan Ernst f833114492 Fix g1gc bootstrap check to only try parsing java version for java 8 2016-11-07 15:05:18 -08:00
Jason Tedor 7c31e8cc58 Modify Javadoc line length in OsProbe.java
This commit changes the Javadocs in OsProbe.java to take advantage of
the fact that the line-length limit is 140 characters. This change makes
these Javadocs easier to read and easier to maintain.
2016-11-07 17:26:28 -05:00
Jason Tedor 6a6e1bed55 Remove JVMCheck
This commit removes JVMCheck. Previously there were three checks in this
class:
 - check for super word bug in JDK 7
 - check for G1GC bugs in JDK 8
 - check for broken IBM JDKs

The first check is obsolete since we require JDK 8 now. The second check
is refactored into a bootstrap check. The third check is removed since
we do not even support the IBM JDK.

Relates #21389
2016-11-07 16:35:39 -05:00
Ryan Ernst edb9d8fc03 Docs: Update discovery plugins example configuration with discovery.zen.hosts_provider (#21390)
This new setting was added with deguicing unicast hosts providers, but
the docs were not updated.
2016-11-07 13:33:08 -08:00
Jason Tedor 1f0c6cd137 Remove double space in bootstrap checks docs
This commit removes a double space in the G1GC check section of the
bootstrap check docs.
2016-11-07 16:28:04 -05:00
Boaz Leskes 68a0d20ccd increase logging level to DEBUG in backwards-5.0 tests 2016-11-07 22:21:47 +01:00
Jason Tedor b30732c464 Migrate G1GC JVM check to bootstrap check
This commit fixes an assertion in G1GCCheck#jvmVersion that was
mistakenly asserting on itself.

Relates #21388
2016-11-07 16:19:05 -05:00
Jason Tedor f80aa65fe9 Remove JDK 7 checks from JVMCheck
This commit removes checks for buggy JDK 7 JVMs from the JVM check as
Elasticsearch does not support JDK 7.

Relates #21381
2016-11-07 15:50:06 -05:00
Ali Beyad 871a077ac4 Fixes get snapshot duplicates when asking for _all (#21340)
Before, when requesting to get the snapshots in a repository, if `_all` was
specified for the set of snapshots to get, any in-progress snapshots would
be returned twice.  This commit fixes the issue ensuring that each snapshot,
whether in-progress or completed, is only returned once when making a call to
get snapshots (GET /_snapshot/{repository_name}/_all).

This commit also enables `_current` to appear anywhere in 
the get snapshots list, and will be used as a pseudo regex to
mean "match any currently running snapshots".

Closes #21335
2016-11-07 14:20:28 -05:00
Jason Tedor 8067412983 Remove load average leniency
Today when acquiring load average stats, this method is rather lenient
when reading /proc/loadavg. This commit removes this leniency so that
any such issues are more likely to be exposed to the end-user.

Relates #21380
2016-11-07 14:04:04 -05:00
Jason Tedor 23a271f092 Address race condition in HTTP pipeline tests
This commit adapts a previous fix to the HTTP pipeline tests for Netty 4
to Netty 3.

Relates #19845
2016-11-07 13:20:22 -05:00
Adrien Grand 263af27d76 Fix docs example after #21218. 2016-11-07 14:57:20 +01:00
Clinton Gormley a36465d479 Removed left over ID and [float] from the setup docs 2016-11-07 14:48:50 +01:00
Clinton Gormley 8383af6734 The routing query string param is supported by mget but was missing from the rest spec (#21357) 2016-11-07 14:14:52 +01:00
Alexander Reelsen 5413efc570 Packaging Deb: configure start-stop-daemon to not go into background (#21343)
On ubuntu 14.04, which uses upstart, where as our debian package uses
sysvinit, there is no stdout/stderr message printed when starting up,
because the start-stop-daemon swallows it.

As Elasticsearch is started to daemonize, we can remove the background
flag from the start-stop-daemon and thus see, if the system does not have
enough memory for starting up - something that happens often on VMs, since
Elasticsearch 5.0 uses 2gb by default instead of one.

Relates #21300
Relates #12716
2016-11-07 13:59:11 +01:00
Adrien Grand 457c87affb Fix PercentilesBucketIT expectations. 2016-11-07 13:48:04 +01:00
Alexander Kazakov c92b4b30a0 Percentiles bucket fails for 100th percentile (#21218)
It mistakenly multiplies the quantile by the array `length` instead of `length-1`.
2016-11-07 13:02:09 +01:00
markharwood dd21aa41be Docs fix - Diversified sampler agg had incorrect title and example
Closes #21347
2016-11-07 10:46:22 +00:00
Nik Everett 593d47efe2 Make it clear _suggest doesn't support source filtering (#21268)
We plan to deprecate `_suggest` during 5.0 so it isn't worth fixing
it to support the `_source` parameter for `_source` filtering. But we
should fix the docs so they are accurate.

Since this removes the last non-`// CONSOLE` line in
`completion-suggest.asciidoc` this also removes it from the list of
files that have non-`// CONSOLE` docs.

Closes #20482
2016-11-06 20:15:45 -05:00
Clinton Gormley 9d56c1b766 Groovy is no longer the default scripting language
Closes #21208
2016-11-05 16:31:34 +01:00
Loek van Gool 1a23739211 Update store.asciidoc (#21353)
* Update store.asciidoc

* Update store.asciidoc

* Update store.asciidoc
2016-11-05 14:58:16 +01:00
Clinton Gormley c970aedd44 Add remote debugger instructions to TESTING.asciidoc 2016-11-05 14:24:57 +01:00
Nik Everett 7ff9ba1604 Fix asciidoc structure for sliced reindex
Asciidoc likes headings just so and will complain and fail the
docs build without it.

Related to #20767
2016-11-04 21:59:19 -04:00
Nik Everett a13a050271 Add automatic parallelization support to reindex and friends (#20767)
Adds support for `?slices=N` to reindex which automatically
parallelizes the process using parallel scrolls on `_uid`. Performance
testing sees a 3x performance improvement for simple docs
on decent hardware, maybe 30% performance improvement
for more complex docs. Still compelling, especially because
clusters should be able to get closer to the 3x than the 30%
number.

Closes #20624
2016-11-04 20:59:15 -04:00
Jason Tedor 555084a226 Write -1 on unbounded queue in cat thread pool
Today when writing an unbounded queue_size in the cat thread pool API,
we write null. This commit modifies this so that the output is -1 so
that the output is always present, and always a numeric value.

Relates #21342
2016-11-04 15:17:55 -04:00
Luca Cavanna c2160a88b5 Remove support for controversial ignore_unavailable and allow_no_indices from indices exists api (#20712)
Exist requests are supposed to never throw an exception, but rather return true or false depending on whether some resource exists or not. Indices exists does that for indices and accepts wildcard expressions too. The way the api works internally is by resolving indices and catching IndexNotFoundException: if an exception is thrown the index does not exist hence it returns false, otherwise it returns true. That works ok only if ignore_unavailable and allow_no_indices indices options are both set to false, meaning that they are strict and any missing index or wildcard expressions that resolves to no indices will lead to an exception that can be thrown and cause false to be returned.

Unfortunately the indices options have  been configurable up until now for this request, meaning that one can set ignore_unavailable or allow_no_indices to true and have the indices exist request return true for indices that really don't exist, which makes very little sense in the context of this api.

This commit removes the indicesOptions setter from the IndicesExistsRequest and makes settable only expandWildcardsOpen and expandWildcardsClosed, hence a subset of the available indices options. This way we can guarantee more consistent behaviour of the indices exists api. We can then remove the ignore_unavailable and allow_no_indices option from indices exists api spec
2016-11-04 19:26:37 +01:00
Jason Tedor f16c308efd Assert status logger does not warn on Log4j usage
Today if you start Elasticsearch with the status logger configured to
the warn level, or use a transport client with the default status logger
level, you will see warn messages about deprecation loggers being
created with different message factories and that formatting might be
broken. This happens because the deprecation logger is constructed using
the message factory from its parent, an artifact leftover from the first
Log4j 2 implementation that used a custom message factory. When that
custom message factory was removed, this constructor invocation should
have been changed to not explicitly use the message factory from the
parent. This commit fixes this invocation. However, we also had some
status checking to all tests to ensure that there are no warn status log
messages that might indicate a configuration problem with Log4j 2. These
assertions blow up badly without the fix for the deprecation logger
construction, and also caught a misconfiguration in one of the logging
tests.

Relates #21339
2016-11-04 14:19:59 -04:00
Andreas Creten 1ab4e0cba4 Add elasticsearcher for PHP (#21143)
Disclaimer: it is a library written by one of the engineers in my company.
2016-11-04 17:27:31 +01:00
Jayson Minard a978f13660 Fix cat thread pool API spec
This commit fixes an issue with the cat thread pool API spec. Namely,
the name of the variable for the thread pool patterns parameter was
misnamed in the spec.

Relates #21332
2016-11-04 12:25:11 -04:00
Simon Willnauer e042536082 Fix SpanMultiTermQueryBuilderTest by using a mock query 2016-11-04 16:00:53 +01:00
Nik Everett a4b3a95f5a Move flush in _cat/indices docs tests (#21117)
Moves the `_flush` in the `_cat/indices` snippets testing framework
to the very first test. We need to flush super early because index
size is cached for a few seconds so we really need to read a
consistent size on the first read so we can sort by it properly.

Closes #21062
2016-11-04 10:32:07 -04:00
Simon Willnauer 436ba7b5fc Validate the `_rollover` target index name early to also fail if dry_run=true (#21330)
Today we validate the target index name late and therefore don't fail for instance
if the target index already exists and `dry_run=true` was specified. This change
validates the index name before we early terminate if dry_run is set.

Closes #21149
2016-11-04 13:53:28 +01:00
Simon Willnauer ed2865863c Remove LateParsingQuery to prevent timestamp access after context is frozen (#21328)
Today we still have a leftover from older percolators where lucene
query instances where created ahead of time and rewritten later.
This `LateParsingQuery` was resolving `now()` when it's really used which we
don't need anymore. As a side-effect this failed to execute some highlighting
queries when they get rewritten since at that point `now` access it not permitted
anymore to prevent bugs when queries get cached.

Closes #21295
2016-11-04 13:53:03 +01:00
Lee Hinman 6666fb1614 Add "all field" execution mode to query_string query
This commit introduces a new execution mode for the query_string query, which
is intended down the road to be a replacement for the current _all field.

It now does auto-field-expansion and auto-leniency when the following criteria
are ALL met:

    The _all field is disabled
    No default_field has been set in the index settings
    No default_field has been set in the request
    No fields are specified in the request

Additionally, a user can force the "all-like" execution by setting the
all_fields parameter to true.

When executing in all field mode, the query_string query will look at all the
fields in the mapping that are not metafields and can be searched, and
automatically expand the list of fields that are going to be queried.

Relates to #19784
2016-11-04 05:46:18 -06:00
Christoph Büscher 6acbefe3f7 Add tests for alternative ways of writing zero offset timezones
According to ISO 8601, a time zone offset of zero, can be stated numerically as
"+00:00", "+0000", or "00". The Joda library also seems to allow for "-00:00",
"-00" and "-0000". This adds some test to the DateMathParserTests that check
that we also conform to this.

Closes #21320
2016-11-04 12:24:40 +01:00
Christoph Büscher f4594d4302 Removing plugin that isn't installed shouldn't trigger usage information
The usage information for `elasticsearch-plugin` is quiet verbose and makes the
actual error message that is shown when trying to remove an non-existing plugin
hard to spot. This changes the error code to not trigger printing the usage
information.

Closes #21250
2016-11-04 11:46:20 +01:00