Commit Graph

21290 Commits

Author SHA1 Message Date
Simon Willnauer 8b201fda46 Merge pull request #17106 from s1monw/fix_can_delete
IndicesStore checks for `allocated elsewhere` for every shard not alocated on the local node

On each cluster-state update we check on the local node if we can delete some shards content.
For this we linearly walk all shards and check if they are allocated and started on another node
and if we can delete them locally. if we can delete them locally we go and ask other nodes if we can
delete them and then if the shared IS active elsewhere issue a state update task to delete it. Yet,
there is a bug in IndicesService#canDeleteShardContent which returns `true` even if that shards
datapath doesn't exist on the node which causes tons of unnecessary node to node communication and
as many state update task to be issued. This can have large impact on the cluster state processing
speed. 

**NOTE:** This only happens for shards that have at least one shard allocated on the node ie. if an `IndexService` exists.
2016-03-15 12:59:45 +01:00
Simon Willnauer 10333e2f05 IndicesStore checks for `allocated elsewhere` for every shard not allocated on the local node
On each clusterstate update we check on the local node if we can delete some shards content.
For this we linearly walk all shards and check if they are allocated and started on another node
and if we can delete them locally. if we can delete them locally we go and ask other nodes if we can
delete them and then if the shared IS active elsewhere issue a state update task to delete it. Yet,
there is a bug in IndicesService#canDeleteShardContent which returns `true` even if that shards
datapath doesn't exist on the node which causes tons of unnecessary node to node communciation and
as many state update task to be issued. This can have large impact on the cluster state processing
speed.

**NOTE:** This only happens for shards that have at least one shard allocated on the node
ie. if an `IndexService` exists.

Closes #17106
2016-03-15 12:58:36 +01:00
Simon Willnauer e8152d375e Try to renew sync ID if `flush=true` on forceMerge
Today we do a force flush which wipes the sync ID if there is one which
can cause the lost of all benefits of the sync ID ie. fast recovery.
This commit adds a check to renew the sync ID if possible. The flush call
is now also not forced since the IW will show pending changes if the forceMerge added new segments.
if we keep using force we will wipe the sync ID even if no renew was actually needed.

Closes #17019
2016-03-15 12:50:57 +01:00
Christoph Büscher bc84cdfed1 Using SortMode enum in all sort builders 2016-03-15 12:43:19 +01:00
Christoph Büscher b4b874f0d8 Merge branch 'master' into feature-suggest-refactoring 2016-03-15 12:11:39 +01:00
Isabel Drost-Fromm be09778dc0 Fix compile error
that's what you get for a "let me quickly try something out here"
2016-03-15 11:12:38 +01:00
Christoph Büscher 1f1f6861b7 Merge branch 'master' into sort-serialization-scriptsort
Conflicts:
	core/src/test/java/org/elasticsearch/search/sort/AbstractSortTestCase.java
2016-03-15 11:03:28 +01:00
Isabel Drost-Fromm 4b5783bdd9 Merge pull request #16127 from MaineC/enhancement/15178-sort-fieldsortbuilder
Refactor FieldSortBuilder to add serialization support
2016-03-15 10:48:20 +01:00
Simon Willnauer e85d8b3817 Merge pull request #17103 from s1monw/followup_15853
Reenable CreateIndexIT#testCreateAndDeleteIndexConcurrently

Since #16442 is merged we should be able to reenable this test as a followup
of #15853 - all issues blocking it have been resolved I guess.
2016-03-15 09:46:22 +01:00
Simon Willnauer 29268f100b Reenable CreateIndexIT#testCreateAndDeleteIndexConcurrently
Since #16442 is merged we should be able to reenable this test as a followup
of #15853 - all issues blocking it have been resolved I guess.
2016-03-15 09:31:13 +01:00
David Pilato 84c862b825 Merge remote-tracking branch 'origin/master' 2016-03-15 09:25:26 +01:00
David Pilato a3bf57d116 Upgrade azure SDK to 0.9.3
We are ATM using azure SDK 0.9.0.

Azure latest release is now 0.9.3 (released in February 2016).

<img width="1024" alt="the central repository search engine google chrome aujourd hui at 08 41 12" src="https://cloud.githubusercontent.com/assets/274222/13662836/a806ba3a-e69d-11e5-8655-4a838db2ef47.png">

Artifacts are on [maven central](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.microsoft.azure%22%20AND%20(a%3Aazure-serviceruntime%20OR%20a%3Aazure-servicebus%20OR%20a%3Aazure-svc-*))

Change log:

## 2016.2.18 Version 0.9.3

* Fix enum bugs in azure-svc-mgmt-websites

## 2016.1.26 Version 0.9.2

* Fix HTTP Proxy for Apache HTTP Client in Service Clients
* Key Vault: Fix KeyVaultKey to not attempt to load RSA Private Key

## 2016.1.8 Version 0.9.1

* Support HTTP Proxy
* Fix token expiration issue #557
* Service Bus: Add missing attributes: partitionKey, viaPartitionKey
* Traffic Manager: Update API version, add MinChildEndpoints for NestedEndpoints
* Media: Add support for Widevine (DRM) dynamic encryption

Closes #17042.
2016-03-15 09:18:34 +01:00
Simon Willnauer 345e988bbc Merge pull request #17072 from s1monw/add_backwards_rest_tests
Add infrastructure to run REST tests on a multi-version cluster

This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Given the amount of problems this change already found I think it's worth having this run with our test suite by default. The structure of this infra will likely change over time but for now it's a step into the right direction. We will likely want to split it up into integTests and integBwcTests etc. so each plugin can have it's own bwc tests but that's left for future refactoring.
2016-03-15 09:17:43 +01:00
Areek Zillur c3078f4d65 adapt tests to use index uuid as folder name 2016-03-14 23:24:24 -04:00
Areek Zillur 2b18a3ce1d use index uuid as folder name to decouple index folder name from index name 2016-03-14 23:24:19 -04:00
Areek Zillur 3daa83b2d2 remove redundant getters in MetaData 2016-03-14 23:24:12 -04:00
Areek Zillur 35f7cfb6c0 Add upgrader to upgrade old indices to new naming convention 2016-03-14 23:24:05 -04:00
Simon Willnauer 554bf2c282 [TEST] Test that all processors are available 2016-03-14 22:35:25 +01:00
Ali Beyad f527a034a3 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-03-14 17:25:02 -04:00
Simon Willnauer 6f28c173e2 [TEST] Test that all processors are available 2016-03-14 21:42:37 +01:00
Jack Conradson dfec4547ea Added one minor comment for expressions tests. 2016-03-14 13:19:52 -07:00
Simon Willnauer bd96075f7f Merge branch 'master' into add_backwards_rest_tests 2016-03-14 21:18:42 +01:00
Simon Willnauer ce6ec511b9 Move shared directory under build/cluster/shared to prevent granting r/w permission to the build/cluster directory 2016-03-14 21:16:44 +01:00
Jack Conradson 7665c4fe6f Merge branch 'master' into pr/17091 2016-03-14 12:16:21 -07:00
Christoph Büscher 17a420e6aa Adressing review comments, adding parsing tests 2016-03-14 15:24:49 +01:00
Christoph Büscher 40f3501d7f Merge branch 'master' into feature-suggest-refactoring 2016-03-14 14:57:57 +01:00
Simon Willnauer e4bed0c97e Improve validation error message on PutMappingRequest 2016-03-14 14:57:43 +01:00
Simon Willnauer 0ebcef0bb4 wrap line after 140 chars 2016-03-14 14:51:43 +01:00
Simon Willnauer 9f382da5d3 Add better validation error message and a dedicated test 2016-03-14 14:27:35 +01:00
Christoph Büscher 3fe07a9754 Adding enum for SortMode and use it in ScriptSortBuilder 2016-03-14 14:18:18 +01:00
Isabel Drost-Fromm 744de1f6cb Throw ParsingExceptions instead of IllegalArgument
... keeps track of the position of the error.
2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm fb647e9bf4 Add sanity checks and support for reverse in FieldSortBuilder.parse(...)
After another round of input from @cbuescher this adds a few more sanity
checks to request parsing.

In addition adds (back) support for the reverse option.
2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm 02e698bc43 Third round of comments 2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm 720f47e87f Second round of comments 2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm dd6e835e30 First comments. 2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm 5bff6e4218 Refactor FieldSortBuilder
* adds json parsing,
* refactors json serialisation,
* adds writable parsing and serialisation,
* adds json and writable roundtrip test
2016-03-14 14:02:22 +01:00
Clinton Gormley 0ed0fea558 Updated link to Joda time zones 2016-03-14 12:24:58 +01:00
Christoph Büscher f0074668db Merge branch 'master' into sort-serialization-scriptsort 2016-03-14 12:15:53 +01:00
Simon Willnauer a69d44ce5c Merge pull request #17048 from s1monw/use_concrete_index_on_clusterstate_update
Resolve index names to Index instances early

Today index names are often resolved lazily, only when they are really
needed. This can be problematic especially when it gets to mapping updates
etc. when a node sends a mapping update to the master but while the request
is in-flight the index changes for whatever reason we would still apply the update
since we use the name of the index to identify the index in the clusterstate.
The problem is that index names can be reused which happens in practice and sometimes
even in a automated way rendering this problem as realistic.
In this change we resolve the index including it's UUID as early as possible in places
where changes to the clusterstate are possible. For instance mapping updates on a node use a
concrete index rather than it's name and the master will fail the mapping update iff
the index can't be found by it's <name, uuid> tuple.

Closes #17048
2016-03-14 11:40:22 +01:00
Christoph Büscher 97638c95fc Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-14 11:13:47 +01:00
Simon Willnauer 31740e279f Resolve index names to Index instances early
Today index names are often resolved lazily, only when they are really
needed. This can be problematic especially when it gets to mapping updates
etc. when a node sends a mapping update to the master but while the request
is in-flight the index changes for whatever reason we would still apply the update
since we use the name of the index to identify the index in the clusterstate.
The problem is that index names can be reused which happens in practice and sometimes
even in a automated way rendering this problem as realistic.
In this change we resolve the index including it's UUID as early as possible in places
where changes to the clusterstate are possible. For instance mapping updates on a node use a
concrete index rather than it's name and the master will fail the mapping update iff
the index can't be found by it's <name, uuid> tuple.

Closes #17048
2016-03-14 11:08:48 +01:00
Clinton Gormley c3cd8564df Corrected regexp syntax docs for COMPLEMENT 2016-03-14 10:46:31 +01:00
Alexander Kazakov 8e6b2b3909 Check that _value is used in aggregations script before setting value to specialValue #14262 2016-03-14 12:04:06 +03:00
Clinton Gormley c90b4f3bae Docs: Added note about upgrading from 1.x to 5.x 2016-03-14 09:58:46 +01:00
Adrien Grand 071b396306 Fix test bug: norms are on by default on _all. 2016-03-14 09:06:11 +01:00
Adrien Grand c50c5a52d5 Rework norms parameters for 5.0. #16987
Changes:
 - no more option to configure eager/lazy loading of the norms (useless now
   that orms are disk-based)
 - only the `string`, `text` and `keyword` fields support the `norms` setting
 - the `norms` setting takes a boolean that decides whether norms should be
   stored in the index but old options are still supported to give users time
   to upgrade
 - setting a `boost` no longer implicitely enables norms (for new indices only,
   this is still needed for old indices)
2016-03-14 08:42:35 +01:00
Adrien Grand 5596e31068 Upgrade to lucene-6.0.0-f0aa4fc. #17075 2016-03-14 07:58:52 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Jason Tedor 8ac5a98b87 Remove links to nonexistent migration docs 2016-03-13 19:12:06 -04:00
Jason Tedor 4353b2e024 Do not pass double-dash arguments on startup
This commit addresses an issue in the init scripts which are passing
invalid command line arguments to the startup script.

Closes #17087
2016-03-13 18:58:08 -04:00