Commit Graph

27032 Commits

Author SHA1 Message Date
Ali Beyad 480cfe3fe0 Fixes snapshot status on failed snapshots (#23833)
If a snapshot is taken on multiple indices, and some of them are "good"
indices that don't contain any corruption or failures, and some of them
are "bad" indices that contain missing shards or corrupted shards, and
if the snapshot request is set to partial=false (meaning don't take a
snapshot if there are any failures), then the good indices will not be
snapshotted either.  Previously, when getting the status of such a
snapshot, a 500 error would be thrown, because the snap-*.dat blob for
the shards in the good index could not be found.

This commit fixes the problem by reporting shards of good indices as
failed due to a failed snapshot, instead of throwing the
NoSuchFileException.

Closes #23716
2017-04-06 20:54:21 -04:00
Nik Everett 1ad5398af7 Fix compilation in Eclipse (#23954)
Eclipse seems to have trouble with CrudIT. It throws an NPE back
to the user. I've filed a bug:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=514884
but for now we should work around it.
2017-04-06 20:40:55 -04:00
Ali Beyad 4f121744bd Removes the retry mechanism from the S3 blob store (#23952)
Currently, both the Amazon S3 client provides a retry mechanism, and the
S3 blob store also attempts retries for failed read/write requests.
Both retry mechanisms are controlled by the
`repositories.s3.max_retries` setting.  However, the S3 blob store retry
mechanism is unnecessary because the Amazon S3 client provided by the
Amazon SDK already handles retries (with exponential backoff) based on
the provided max retry configuration setting (defaults to 3) as long as
the request is retryable.  Hence, this commit removes the unneeded retry
logic in the S3 blob store and the S3OutputStream.

Closes #22845
2017-04-06 19:58:53 -04:00
jaymode 53e3ddf2f0
Test: remove test that will never run on master
This test was added in #23950 for backporting and review, but it is always skipped on master so
this commit deletes it.
2017-04-06 15:50:08 -04:00
Jay Modi 495bf21b46 Preserve response headers when creating an index (#23950)
This commit preserves the response headers when creating an index and updating settings for an
index.

Closes #23947
2017-04-06 20:38:09 +01:00
Jason Tedor 2620200ff7 Ensure output for createClone task
This commit changes the task type from LoggedExec to Exec for the BWC
createClone task to ensure that we can get some debug output.
2017-04-06 08:37:04 -04:00
Nik Everett 048191ceb6 CONSOLEify highlighting a function_score docs
Converts many of the partial examples into full search requests.

Relates #18160
2017-04-06 08:13:56 -04:00
Jim Ferenczi 042f7566e8 update Version.V_5_3_1_UNRELEASED to the latest bugfix release of Lucene:6_4_2 2017-04-06 10:03:17 +02:00
Jim Ferenczi 38009efedd Disable graph analysis at query time for shingle and cjk filters producing tokens of different size (#23920)
This change disables graph analysis of token streams containing a shingle or a cjk filters that produce shingle or ngram of different size. The graph analysis is disabled for phrase and boolean queries.

Closes #23918
2017-04-06 08:55:00 +02:00
Ryan Ernst 203f8433c2 Collapse packages in ec2 discovery plugin (#23909)
This commit collapses all the classes inside ec2 discovery to a single
package name.
2017-04-05 23:51:49 -07:00
Nik Everett 471af48170 Docs: fix response object name
In #22810 I renamed a response object used by reindex and friends
but didn't update the java-api documentation that uses it. This
makes that update.
2017-04-05 16:03:49 -04:00
Nik Everett 0b20a59391 Docs test: defend against round numbers
If a shard has a nice, round number the test in the `_cat/shards`
reference file would fail. They should be ok with it. A failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.3+multijob-unix-compatibility/os=fedora/93/console
2017-04-05 15:31:11 -04:00
Ryan Ernst c48b60dcf1 Tests: Extend rest test timeout to 30 minutes for windows (#23912)
Windows rest tests consistenly fail because the filesystem appears to be
an order of magnitude slower than that of *nix, at least in the context
of our rest tests. This commit overrides the suite timeout to 30 mins
for windows. From past failures, it appears this should be enough, as
the tests seem to fail when they are almost complete. The default suite
timeout for ESTestCase is 20 mins, so this leaves ample buffer for
windows shenanigans.
2017-04-05 12:11:03 -07:00
Tim Brooks 5b1fbe5e6c Decouple BulkProcessor from client implementation (#23373)
This commit modifies the BulkProcessor to be decoupled from the
client implementation. Instead it just takes a
BiConsumer<BulkRequest, ActionListener<BulkResponse>> that executes
the BulkRequest.
2017-04-05 12:12:43 -05:00
Lee Hinman 0257a7b97a Only re-parse operation if a mapping update was needed
When executing an index operation on the primary shard,
`TransportShardBulkAction` first parses the document, sees if there are any
mapping updates that needs to be applied, and then updates the mapping on the
master node. It then re-parses the document to make sure that the mappings have
been applied and propagated.

This adds a check that skips the second parsing of the document in the event
there was not a mapping update applied in the first case.

Fixes a performance regression introduced in #23665
2017-04-05 09:29:44 -06:00
Adrien Grand d5d0f140d6 The `filter` and `significant_terms` aggregations should parse the `filter` as a filter, not as a query. (#23797)
This is important for some queries like `bool`, which are parsed differently
depending on whether we want to get a query or a filter.
2017-04-05 16:46:21 +02:00
Simon Willnauer adccdbb3cf Simplify sorted top docs merging in SearchPhaseController (#23881)
Today we have several code paths to merge top docs based on the number of
search results returned from the shards. If there is a only a single shard
holding any hits we go a different code path with quite some complexity while
if there are more than one the code is basically duplicated to safe the
creation of a dense array of top docs which can be large if there are many results.
This commit removes the need of the dense array and in-turn the justification for
the optimization. This commit introduces a single code path to merge top docs.
2017-04-05 14:49:35 +02:00
Boaz Leskes 75b4f408e0 Refactor InternalEngine's index/delete flow for better clarity (#23711)
The InternalEngine Index/Delete methods (plus satellites like version loading from Lucene) have accumulated some cruft over the years making it hard to clearly the code flows for various use cases (primary indexing/recovery/replicas etc). This PR refactors those methods for better readability. The methods are broken up into smaller sub methods, albeit at the price of less code I reused.

To support the refactoring I have considerably beefed up the versioning tests.

This PR is a spin-off from #23543 , which made it clear this is needed.
2017-04-05 14:43:01 +02:00
Boaz Leskes c89fdd938e ZenDiscovery - only validate min_master_nodes values if local node is master (#23915)
The purpose of this validation is to make sure that the master doesn't step down
due to a change in master nodes, which also means that there is no way to revert
an accidental change. Since we validate using the current cluster state (and
not the one from which the settings come from) we have to be careful and only
validate if the local node is already a master. Doing so all the time causes
subtle issues. For example, a node that joins a cluster has no nodes in its
current cluster state. When it receives a cluster state from the master with
a dynamic minimum master nodes setting int it, we must make sure we don't reject it.

Closes #23695
2017-04-05 14:31:32 +02:00
Jason Tedor 24127bf416 Remove hardcoded ports from SingleNodeDiscoveryIT
SingleNodeDiscoveryIT uses a hardcoded port for the purpose of binding
two nodes within the limited port range that an unconfigured unicast zen
ping hosts list would try to discover another node on. This commit at
least removes this hardcoding for the first node to come up, although
still tries to bind the second node to the limited port range after the
first node has bound.
2017-04-05 08:17:33 -04:00
Suhas Karanth 777b5a3c16 Correct documentation for Min Bucket Aggregation (#23867) 2017-04-05 12:39:37 +02:00
Jayant Ameta 47160ba6ed Update secure-settings.asciidoc (#23891) 2017-04-05 11:18:23 +02:00
Luca Cavanna 318d365b12 [TEST] make sure that fromXContent doesn't rely on keys ordering (#23901)
We shuffle the keys before we parse our responses for the high level client so that we make sure we never rely on keys ordering.
2017-04-05 11:12:34 +02:00
Jason Tedor afd45c1432 Revert "Closing a ReleasableBytesStreamOutput closes the underlying BigArray (#23572)"
This reverts commit 6bfecdf921.
2017-04-04 20:33:51 -04:00
Ryan Ernst d31d2caf09 Collapse packages in repository-s3 (#23907)
This commit puts all the classes in the repository-s3 plugin into a
single package.  In addition to simplifying the plugin, it will make it
easier to test as things that should be package private will not be
difficult to use inside tests alone.
2017-04-04 15:15:25 -07:00
Alex Chan 7f43b75d2b Add a missing apostrophe (let us ~> let's) (#23905) 2017-04-04 14:58:32 -06:00
Jay Modi 6bfecdf921 Closing a ReleasableBytesStreamOutput closes the underlying BigArray (#23572)
This commit makes closing a ReleasableBytesStreamOutput release the underlying BigArray so
that we can use try-with-resources with these streams and avoid leaking memory by not returning
the BigArray. As part of this change, the ReleasableBytesStreamOutput adds protection to only release the BigArray once.

In order to make some of the changes cleaner, the ReleasableBytesStream interface has been
removed. The BytesStream interface is changed to a abstract class so that we can use it as a
useable return type for a new method, Streams#flushOnCloseStream. This new method wraps a
given stream and overrides the close method so that the stream is simply flushed and not closed.
This behavior is used in the TcpTransport when compression is used with a
ReleasableBytesStreamOutput as we need to close the compressed stream to ensure all of the data
is written from this stream. Closing the compressed stream will try to close the underlying stream
but we only want to flush so that all of the written bytes are available.

Additionally, an error message method added in the BytesRestResponse did not use a builder
provided by the channel and instead created its own JSON builder. This changes that method to use the channel builder and in turn the bytes stream output that is managed by the channel.
2017-04-04 17:01:30 +01:00
Jason Tedor 3136ed1490 Rename random ASCII helper methods
This commit renames the random ASCII helper methods in ESTestCase. This
is because this method ultimately uses the random ASCII methods from
randomized runner, but these methods actually only produce random
strings generated from [a-zA-Z].

Relates #23886
2017-04-04 11:04:18 -04:00
Jason Tedor a01f77210a Fix Javadocs for BootstrapChecks#enforceLimits
This commit adds a description for a parameter that was added to
BootstrapChecks#enforceLimits(BoundTransportAddress, String) without the
Javadocs having been updated.
2017-04-04 09:42:19 -04:00
Jason Tedor 51b5dbffb7 Disable bootstrap checks for single-node discovery
While there are use-cases where a single-node is in production, there
are also use-cases for starting a single-node that binds transport to an
external interface where the node is not in production (for example, for
testing the transport client against a node started in a Docker
container). It's tricky to balance the desire to always enforce the
bootstrap checks when a node might be in production with the need for
the community to perform testing in situations that would trip the
bootstrap checks. This commit enables some flexibility for these
users. By setting the discovery type to "single-node", we disable the
bootstrap checks independently of how transport is bound. While this
sounds like a hole in the bootstrap checks, the bootstrap checks can
already be avoided in the single-node use-case by binding only HTTP but
not transport. For users that are genuinely in production on a
single-node use-case with transport bound to an external use-case, they
can set the system property "es.enable.bootstrap.checks" to force
running the bootstrap checks. It would be a mistake for them not to do
this.

Relates #23598
2017-04-04 09:39:04 -04:00
Jim Ferenczi c14be20744 Add unit tests for the missing aggregator (#23895)
* Add unit tests for the missing aggregator

Relates #22278
2017-04-04 14:37:33 +02:00
Jim Ferenczi a04350f0dd Add a property to mark setting as final (#23872)
This change adds a setting property that sets the value of a setting as final.
Updating a final setting is prohibited in any context, for instance an index setting
marked as final must be set at index creation and will refuse any update even if the index is closed.
This change also marks the setting `index.number_of_shards` as Final and the special casing for refusing the updates on this setting has been removed.
2017-04-04 12:35:48 +02:00
Boaz Leskes 2266947ac5 testDifferentRolesMaintainPathOnRestart - fix broken comment 2017-04-04 11:03:44 +02:00
Boaz Leskes 20b274d7b9 testDifferentRolesMaintainPathOnRestart - lower join timeout as split elections are likely
the test reduce the wait for initial cluster state to 0, causing multiple nodes to be start while elections are going on. This means there is a chance of a split election which shouldn't cause the test to time out.
2017-04-04 10:36:09 +02:00
Jason Tedor 71293a89bf Introduce single-node discovery
This commit adds a single node discovery type. With this discovery type,
a node will elect itself as master and never form a cluster with another
node.

Relates #23595
2017-04-04 03:02:58 -04:00
Jason Tedor 3bd2efa177 Await termination after shutting down executors
When terminating an executor service or a thread pool, we first
shutdown. Then, we do a timed await termination. If the await
termination fails because there are still tasks running, we then
shutdown now. However, this method does not wait for actively executing
tasks to terminate, so we should again wait for termination of these
tasks before returning. This commit does that.

Relates #23889
2017-04-04 03:01:00 -04:00
Jason Tedor 6234a49fb3 Fix initialization issue in ElasticsearchException
If a test touches ElasticsearchExceptionHandle before the class
initialzer for ElasticsearchException has run, a circular class
initialization problem can arise. Namely, the class initializer for
ElasticsearchExceptionHandle depends on the class initializer for
ElasticsearchExceptionHandle which depends on the class initializer for
all the classes that extend ElasticsearchException, but these classes
can not be loaded because ElasticsearchException has not finished its
class initializer. There are tests that can trigger this before
ElasticsearchException has been loaded due to an unlucky ordering of
test execution. This commit addresses this issue by making
ElasticsearchExceptionHandle private, and then exposing methods that
provide the necessary values from ElasticsearchExceptionHandle. Touching
these methods will force the class initializer for
ElasticsearchException to run first.
2017-04-04 00:33:00 -04:00
wyukawa 3274eab41d Fix bulk queue size in thread pool docs
This commit fixes an incorrect specification for the default queue size
for the bulk thread pool in the thread pool docs.

Relates #23870
2017-04-03 22:39:24 -04:00
Lee Hinman b6b9ef8e26 [DOCS] Remove line about eager loading global ordinals
Fielddata can no longer be configured to be loaded eagerly (it only accepts
`true` and `false`), so this line is a little misleading because it talks about
a procedure we can no longer do.
2017-04-03 12:56:21 -06:00
Boaz Leskes ad6eea92d6 GceDiscoverTests - remove intitial_state_timeout 2017-04-03 16:50:40 +02:00
Boaz Leskes 48b0121f60 SpecificMasterNodesIT shouldn't use autoMinMasterNodes
as it tweaks the `discovery.initial_state_timeout` setting.
2017-04-03 16:23:17 +02:00
Boaz Leskes 40eb68c95a testRestorePersistentSettings doesn't to mess with discovery settings 2017-04-03 16:23:17 +02:00
Boaz Leskes 55a3fd1919 testDifferentRolesMaintainPathOnRestart shouldn't use auto managing of min master nodes
It starts nodes in any order and thus it disabled the wait for first cluster state at node start up time
the later is required for the auto management logic.

Closes #23728
2017-04-03 16:23:17 +02:00
Nik Everett 9d2293b381 CONSOLEify the "using scripts" documentation
I found an error in one of the Painless scripts as part of
the conversion.

Relates to #18160
2017-04-03 10:15:29 -04:00
Colin Goodheart-Smithe 8482503f9b
Adds tests for cardinality and filter aggregations
Relates to #22278
2017-04-03 10:09:27 +01:00
David Pilato 0951e20a87 Merge branch 'pr/22728-azure-backoff' 2017-04-03 10:54:03 +02:00
David Pilato 17be03e85e Add Backoff policy to azure repository
With this commit, Azure repositories are now using an Exponential Backoff policy before failing the backup.
It uses Azure SDK default values for this policy:

* `30s` delta backoff base with
   * `3s` min
   * `90s` max
* `3` retries max

Users can define the number of retries they wish by setting `cloud.azure.storage.xxx.max_retries` where `xxx` is the azure named account.

Closes #22728.
2017-04-03 10:52:44 +02:00
Colin Goodheart-Smithe cad4fcd9c9
Revert "Adds tests for cardinality and filter aggregations (#23826)"
This reverts commit 058869ed54.
2017-04-03 09:45:16 +01:00
Colin Goodheart-Smithe 058869ed54 Adds tests for cardinality and filter aggregations (#23826)
* Adds tests for cardinality and filter aggregations

Relates to #22278

* addresses review comments
2017-04-03 09:39:03 +01:00
Boaz Leskes 5cf1d4ae90 mute testDifferentRolesMaintainPathOnRestart
See https://github.com/elastic/elasticsearch/issues/23728
2017-04-03 10:23:04 +02:00