Commit Graph

22358 Commits

Author SHA1 Message Date
Clinton Gormley 589b6c63c6 Include shrink-index.asciidoc 2016-05-31 11:50:50 +02:00
Simon Willnauer 867d35c084 Never trip circuit breaker in liveness request (#18627)
We don't want transport clients to disconnect due to circuit breakers
preventing the liveness request from executing.

Relates to #17951
2016-05-31 11:31:52 +02:00
Simon Willnauer 502a775a7c Add primitive to shrink an index into a single shard (#18270)
This adds a low level primitive operations to shrink an existing
index into a new index with a single shard. This primitive expects
all shards of the source index to allocated on a single node. Once the target index is initializing on the shrink node it takes a snapshot of the source index shards and copies all files into the target indices data folder. An [optimization](https://issues.apache.org/jira/browse/LUCENE-7300) coming in Lucene 6.1 will also allow for optional constant time copy if hard-links are supported by the filesystem. All mappings are merged into the new indexes metadata once the snapshots have been taken on the merge node.

To shrink an existing index all shards must be moved to a single node (one instance of each shard) and the index must be read-only:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_settings' -d '{
    "settings" : {
        "index.routing.allocation.require._name" : "shrink_node_name",
        "index.blocks.write" : true 
    }
}
```
once all shards are started on the shrink node. the new index can be created via:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_shrink/logs_single_shard' -d '{
    "settings" : {
        "index.codec" : "best_compression",
        "index.number_of_replicas" : 1
    }
}'
```

This API will perform all needed check before the new index is created and selects the shrink node based on the allocation of the source index. This call returns immediately, to monitor shrink progress the recovery API should be used since all copy operations are reflected in the recovery API with byte copy progress etc.

The shrink operation does not modify the source index, if a shrink operation should
be canceled or if the shrink failed, the target index can simply be deleted and
all resources are released.
2016-05-31 10:41:44 +02:00
Ryan Ernst c911f4d951 Merge pull request #18646 from rjernst/jarhell_evil_tests
Tests: Remove unnecessary evil jarhell tests
2016-05-31 00:08:36 -07:00
Adrien Grand 9570feaf8d Process dynamic templates in order. #18638
When calling `findTemplateBuilder(context, currentFieldName, "text", null)`,
elasticsearch ignores all templates that have a `match_mapping_type` set since
no dynamic type is provided (the last parameter, which is null in that case).
So this should only be called _last_. Otherwise, if a path-based template
matches, it will have precedence over all type-based templates.

Closes #18625
2016-05-31 09:00:10 +02:00
Ryan Ernst 454de6a8f2 Tests: Remove unnecessary evil jarhell tests
We have 3 evil tests for jarhell. They have been failing in java 9
because of how evil they are. The first checks the leniency we add for
jarhell in the jdk itself. This is unecessary, since if the leniency
wasn't there, we would already be failing all jarhell checks. The second
is checking the compile version is compatible with the jdk. This is
simpler since we don't need to fake the java version: we know 1.7 should
be compatibile with both java 8 and 9, so we can use that as a constant.
Finally the last test checks if the java version system property is
broken. This is simply something we should not check, we have to trust
that java specifies it correctly, and again, if it was broken, all
jarhell checks would be broken.
2016-05-30 21:39:56 -07:00
Jack Conradson d1f4e2ef04 Merge branch 'master' into each 2016-05-30 16:22:14 -07:00
Boaz Leskes 38bee27b11 testCorruptionOnNetworkLayerFinalizingRecovery should keep on trying to assign replicas
In rare cases it can reach the limit set by index.allocation.max_retries
2016-05-30 21:37:34 +02:00
Simon Willnauer f74a78940c Improve phrase suggest test speed (#18633)
There is no reason to read the entire marvel hero file to test the features,
it might take several seconds to do so which is unnecessary.
This commit also splits SearchSuggestTests into core and modules/mustache 
also add @Nighlty to forbidden API to make sure we don't use it since they won't run in CI these days.
2016-05-30 17:22:03 +02:00
Jason Tedor 04cae88ff4 Do not use Lucene SuppressForbidden
Lucene SuppressForbidden is marked lucene.internal and should not be
used outside of Lucene. This commit removes the uses of this class
within Elasticsearch. Instead,
org.elasticsearch.common.SuppressForbidden should be used, which was
already the case in most places.
2016-05-30 10:57:33 -04:00
Boaz Leskes a1c4086a17 No need to inject the ClusterService into IndicesService (#18639)
It's already available where it's needed.
2016-05-30 16:55:44 +02:00
Britta Weber fc1696822f Fix filtering of node ids for TransportNodesAction (#18634)
* Fix filtering of node ids for TransportNodesAction

Don't mix up member variables  with local variables
in constructor.

closes #18618
2016-05-30 15:26:20 +02:00
Yannick Welsch 148fd3f891 [TEST] mute testDeleteIndexStore test
Issue: https://github.com/elastic/elasticsearch/issues/18558
2016-05-30 11:57:42 +02:00
Tanguy Leroux 16307eee6f Remove extra bin/ directory in bin folder
Recent changes adds an extra bin/ directory that contains Windows bat files in the normal Bin folder of elasticsearch. For exemple the script elasticsearch.bat is located in bin/bin/elasticsearch.bat instead of bin/elasticsearch.bat.
2016-05-30 09:19:35 +02:00
Jason Tedor 37a3588c37 Fix min. master nodes links in boostrap check docs
This commit fixes two links to the minimum master nodes configuration
section of the docs in the bootstrap check docs.
2016-05-29 08:01:16 -04:00
Clinton Gormley e35bd11581 Update bootstrap-checks.asciidoc
Fixed asciidoc
2016-05-29 11:56:02 +02:00
Clinton Gormley b2f2d38ebb Update configuring.asciidoc
Minor asciidoc changes
2016-05-29 11:50:36 +02:00
Robert Muir 7c77c62dbc Merge pull request #18621 from rmuir/java.time
add java.time packages to painless whitelist
2016-05-28 06:51:09 -04:00
Simon Willnauer 32e9a879b4 Use at least 2 docs to see deleted docs
Closes #18623
2016-05-28 08:14:49 +02:00
Jack Conradson f5be0982d9 Added initial infrastructure to allow for each to be made. 2016-05-27 17:41:31 -07:00
Jack Conradson 2d8030f09f Added foreach in grammar. 2016-05-27 13:50:38 -07:00
Robert Muir aa53285448 add java.time packages to painless whitelist 2016-05-27 15:07:27 -04:00
Martijn van Groningen fbd3f8df2b percolator: Don't bail query term extraction when a must clause contains an unsupported query, if there is at least one other must clause in a boolean query that can be extracted. 2016-05-27 19:57:50 +02:00
Jason Tedor 46162a40e7 Additional bootstrap check doc fixes
This commit fixes some additional poorly-formatted internal and external
links in the bootstrap check docs.
2016-05-27 10:58:13 -04:00
David Pilato 871d7411cb Merge pull request #18616 from alexshadow007/patch-1
Update repository-gcs.asciidoc
2016-05-27 16:49:31 +02:00
Alexander Kazakov 8ff0077b52 Update repository-gcs.asciidoc
Fixed link
2016-05-27 17:44:58 +03:00
Jason Tedor df7b3c6e82 Fix translog replay multiple operations same doc
Modifying the translog replay to not replay again into the translog
introduced a bug for the case of multiple operations for the same
doc. Namely, since we were no longer updating the version map for each
operation, the second operation for a doc would be treated as a creation
instead of as an update. This commit fixes this bug by placing these
operations into version map. This commit includes a failing test case.

Relates #18611
2016-05-27 09:43:19 -04:00
Christoph Büscher f0020caf7c Merge pull request #18589: Improve TimeUnitRounding for edge cases and DST transitions 2016-05-27 15:37:20 +02:00
Simon Willnauer 98fd45dc02 Move DocStats under Engine to get more accurate numbers (#18587)
Today we pull doc stats from an index reader which might not reflect reality.
IndexWriter might have merged all deletes away but due to a missing refresh
the stats are completely off. This change pulls doc stats from the IndexWriter
directly instead of relying on refreshes to run regularly. Note: Buffered deletes
are still not visible until the segments are flushed to disk.
2016-05-27 15:31:40 +02:00
Yannick Welsch 2b47a2643c Only fail relocation target shard if failing source shard is a primary (#18574)
If the relocation source fails during the relocation of a shard from one node to another, the
relocation target is currently failed as well. For replica shards this is not necessary,
however, as the actual shard recovery of the relocation target is done via the primary shard.
2016-05-27 15:28:57 +02:00
Jason Tedor 123e40726e Fix bootstrap check docs
This commit fixes some incorrect links in the bootstrap check docs.
2016-05-27 09:19:49 -04:00
Christoph Büscher 29c970958c Adding tests for derivatives on date histogram aggregation with time zones 2016-05-27 15:18:11 +02:00
Christoph Büscher 4439a86cb8 Improve TimeUnitRounding for edge cases and DST transitions
Our current testing for TimeUnitRoundings rounding() and nextRoundingValue()
methods that are used especially for date histograms lacked proper randomization
for time zones. We only did randomized tests for fixed offset time zones
(e.g. +01:00, -05:00) but didn't account for real world time zones with
DST transitions.

Adding those tests revealed a couple of problems with our current rounding logic.
In some cases, usually happening around those transitions, rounding a value down
could land on a value that itself is not a proper rounded value. Also sometimes
the nextRoundingValue would not line up properly with the rounded value of all
dates in the next unit interval.

This change improves the current rounding logic in TimeUnitRounding in two ways:
it makes sure that calling round(date) always returns a date that when rounded
again won't change (making round() idempotent) by handling special cases happening
during dst transitions by performing a second rounding. It also changes the
nextRoundingValue() method to itself rely on the round method to make sure we
always return rounded values for the unit interval boundaries.

Also adding tests for randomized TimeUnitRounding that assert important basic
properties the rounding values should have. For better understanding and readability
a few of the pathological edge cases are also added as a special test case.
2016-05-27 15:18:11 +02:00
Jason Tedor 41710f1028 Upgrade joda-time to 2.9.4
This commit upgrades joda-time to version 2.9.4 to integrate a bug fix
there into Elasticsearch.

Relates #18609
2016-05-27 08:51:19 -04:00
javanna 1a298c2e6f Revert "Revert "Update the minimum required version of gradle to 2.13""
This reverts commit 0b965167fd.
All the CI machines have been updated to gradle 2.13, we should be ok upgrading the required minimum version.
2016-05-27 14:40:38 +02:00
Martijn van Groningen 0e9f3addd2 Nested inner hits shouldn't use relative paths
Like on other places in the query dsl the full field name should be used.
Before this change this wasn't the case for nested inner hits when source filtering was used.
Highlighting has a workaround, which is now removed as the source of nested inner hits can only be refered by the full name.

Closes #16653
2016-05-27 13:41:45 +02:00
javanna 0b965167fd Revert "Update the minimum required version of gradle to 2.13"
This reverts commit 8c6cbf0305. We need to update all our CI machines first, then get this change back in.
2016-05-27 13:16:43 +02:00
Luca Cavanna 5794108506 Merge pull request #18595 from javanna/enhancement/gradle_2.13
Update minimum gradle version and remove assumption around target compatibility
2016-05-27 12:35:00 +02:00
javanna 0e563b3fab Don't assume that we always compile for java 1.8 or 1.9
This change makes it possible to compile a separate project with e.g. targetCompatibility 1.7. Adds specific options (compact profile) only when compiling for >= 1.8.
2016-05-27 12:26:06 +02:00
javanna 8c6cbf0305 Update the minimum required version of gradle to 2.13
It came out with improvements around idea integration and language levels. This will make it possible to have the upcoming java client as a new project compiled against java 7 and have idea working on the right language level.
2016-05-27 12:26:06 +02:00
Jason Tedor cebbf0de41 Do not replay into translog on local recovery
When performing a local recovery, the engine replays operations
recovered from the translog into the translog again. These duplicated
operations are eventually cleared after successful recovery on flush,
but there is no need to play these operations into the translog at
all. This commit modifies the local recovery process so as to not replay
these operations into the translog.

Relates #18547
2016-05-27 06:04:11 -04:00
Jason Tedor 82713bab6d Add bootstrap check docs
This commit adds documentation for the bootstrap checks and provides
either links or inline guidance for setting the necessary settings to
pass the bootstrap checks.

Relates #18605
2016-05-27 06:03:35 -04:00
Boaz Leskes 318a4e3ef6 Introduce dedicated master nodes in testing infrastructure (#18514)
This PR changes the InternalTestCluster to support dedicated master nodes. The creation of dedicated master nodes can be controlled using a new `supportsMasterNodes` parameter to the ClusterScope annotation. If set to true (the default), dedicated master nodes will randomly be used. If set to false,  no master nodes will be created and data nodes will also be allowed to become masters. If active, test runs will either have 1 or 3 masternodes
2016-05-27 08:44:20 +02:00
Igor Motov fb763c1e8e Add ability to store results for long running tasks
The results of the tasks are stored in a special index .results
2016-05-26 19:49:13 -04:00
Robert Muir 3f06d9f3b8 Merge pull request #18600 from rmuir/new_script_exception
replace ScriptException with a better one
2016-05-26 17:51:34 -04:00
Jack Conradson cde83e363a Add an update script equivalent test. 2016-05-26 13:16:22 -07:00
Jason Tedor 7c3715e009 Remove a leftover "es." prefix from docs
This commit removes a leftover usage of the "es." in the CLI syntax in
the zip/targz docs.
2016-05-26 15:03:42 -04:00
Robert Muir 76ca4af561 move instanceof to catch block 2016-05-26 15:03:13 -04:00
Jason Tedor d23db39445 Merge pull request #18594 from jasontedor/plugins-cleanup
Plugins cleanup
2016-05-26 14:46:09 -04:00
Jason Tedor f16f65741e Fix when plugins directory is symlink
This commit fixes an issue with the plugins directory being a symbolic
link. Namely, the install plugins command attempts to always create the
plugins directory just in case it does not exist. The JDK method used
here guarantees that the directory is created, and an exception is not
thrown if the directory could not be created because it already
exists. The problem is that this JDK method does not respect symlinks so
its internal existence checks fails, it proceeds to attempt to create
the directory, but the directory creation fails because the symlink
exists. This is documented as being not an issue. We work around this by
checking if there is a symlink where we expect the plugins directory to
be, and only attempt to create if not. We add a unit test that plugin
installation to a symlinked plugins directory works as expected.
2016-05-26 14:10:32 -04:00