Commit Graph

11032 Commits

Author SHA1 Message Date
Ryan Ernst fe62e463e1 fix test bug leftover from #9914 2015-02-27 12:17:19 -08:00
Boaz Leskes 0cec37f3c3 Recovery: unify RecoveryState management to IndexShard and clean up semantics
We keep track of the current stage of recovery using an instance of RecoveryState which is stored on the relevant IndexShard. At the moment changes to this object are made in many places of the code, which are charged of doing it in the right order, keeping track of timers and many more. Also the changes to shard state are decoupled from the recovery stages which caused #9503.

This PR refactors this and brings all of the changes into IndexShard. It also makes all recovery follow the exact same stages and shortcut some. This is in order to keep things simple and always the same (those shortcuts didn't add anything, we ended doing it all anyway).

Also, all timer management is now folded into RecoveryState and unit tests are added.

This closes #9503 by moving the shard to post recovery only once the recovery is done (before they were decoupled), meaning that master promotion of the target shard to started can not cancel the recovery.

Closes #9902
2015-02-27 21:09:50 +01:00
IsaacHaze ee163e570b Docs: Update snapshots.asciidoc
Adds more determiners.

Closes #9673
2015-02-27 20:42:12 +01:00
Boaz Leskes 0f1c779d2c Engine: close snapshots before recovery counter
#9760 was a fix for translog leaking due to measing a delete flag. This is not needed here as we have a better solution to not loose the flag. This commit takes the changes from 1x in order  to keep the code base similar and enjoy the extra tests.

Closes #9760
2015-02-27 20:22:55 +01:00
Ryan Ernst 702b2abd1b fix compilation issue from forward port of #9916 2015-02-27 11:20:55 -08:00
Ryan Ernst dd0084de83 Mappings: Lock down _ttl field
While the parser allowed changing field type settings, these would never
have been serialized.  So this change simply removes parsing using
parseField. Backcompat will still work if a user uploads old settings
(they just would never have worked anyways, so we continue ignoring
them with 1.x, and 2.x will now error).

see #8143
closes #9914
2015-02-27 11:11:04 -08:00
Ryan Ernst 9d708e20a0 Mappings: Lock down _size field
This also changes the stored setting for _size to true (for
indexes created in 2.x).

see #8143
closes #9913
2015-02-27 11:09:52 -08:00
Ryan Ernst f87fa8172e Tests: Add back tests.cluster
This setting is used by the release script to run rest tests against
the version being released.  It used to work only for tests using
the global cluster.  Now it supercedes both SUITE and TEST scope
test clusters.

closes #9916
2015-02-27 11:08:24 -08:00
Robert Muir 4ca57ab10a Upgrade to Lucene r1662607
Closes #9915.

Squashed commit of the following:

commit cfa59f5a3f03d9d1b432980dcee6495447c1e7ea
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Feb 27 12:10:16 2015 -0500

    add missing null check

commit 62fe5403068c730c0e0b6fd1ab1a0246eeef6220
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Feb 27 11:31:53 2015 -0500

    Disable ExtrasFS for now, until we hook all this in properly in a separate issue.

commit 822795c57c5cf846423fad443c2327c4ed0094ac
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Feb 27 10:12:02 2015 +0100

    Fix PercolatorTests.

commit 98b2a0a7d8298648125c9a367cb7e31b3ec7d51b
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Feb 27 09:27:11 2015 +0100

    Fix ChildrenQueryTests.

commit 9b99656fc56bbd01c9afe22baffae3c37bb48a71
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Feb 26 20:50:02 2015 -0500

    cutover apis, no work on test failures yet.
2015-02-27 12:51:36 -05:00
jaymode efbda318d0 Tests: check node ports availability when using unicast discovery
Some tests failures are seen when a node attempts to use a port that is already bound
by some other process on the test machine. This commit adds a bind to test port availability
and iterates over the port range until an available port is found. This reduces the likelihood
of a test node failing to start up due to the port already being bound.
2015-02-27 11:49:54 -05:00
Simon Willnauer ce53e20351 [CORE] Remove Restart API and remove Node#stop()
The restart API seems pretty esoteric and is entirely untested.
This commit removes it without a replacement.

Closes #9841
2015-02-27 13:58:23 +01:00
Simon Willnauer 2b8827d75c [CORE] Remove component settings from AbstractComponent
Today we have two ways of getting a setting, either with the full settings key or with only
the last part of the key where the prefix is implicit depending on the package the class is in via
component settings. this is trappy as well as confusing for users and can break easily if a class is moved
to a new package since the prefix then implicitly changes.
This commit removes the component settings from the codebase.
2015-02-27 13:01:26 +01:00
Ryan Ernst e221dc20a4 remove thread leak filter, should have been removed in #9843 2015-02-26 23:24:56 -08:00
Ryan Ernst 3b7928d568 Mappings: Lock down _field_names field
Now that we have an explicit `enabled` flag, we can lock down
the field type so it is not mungeable.

see #8143
closes #9912
2015-02-26 15:15:59 -08:00
Ryan Ernst 7181bbde26 Mappings: Remove _boost field
This has been deprecated since 1.0.0.RC1. It is finally removed here.

closes #8875
2015-02-26 15:07:07 -08:00
Ryan Ernst cc8f159a3c fix bwc test to be more lenient on error message when _field_names is
not supported
2015-02-26 14:32:50 -08:00
Ryan Ernst 78df69e6a0 Mappings: Lock down _routing field
`required` is now the only changeable settings (on indexes created after 1.x).

see #8143
closes #9895
2015-02-26 13:09:41 -08:00
Ryan Ernst 257cdcd110 Mappings: Add enabled flag for _field_names to replace disabling through index=no
Almost all of our meta fields that allow enabling/disabling have an `enabled`
setting.  However, _field_names is enabled by default, and disabling
requires setting `index=no`.  This change adds a flag similar to that
with other meta fields.

closes #9893
2015-02-26 13:01:07 -08:00
Simon Willnauer 261e21a386 [TEST] Fix NPE in ElasticsearchIntegrationTest if no indexer is provided
Closes #9907
2015-02-26 21:13:55 +01:00
Nicholas Knize b05f5ebee8 [TEST] Adding 'AwaitsFix' annotation to randomGeoCollection test
Random geo shape testing periodically fails on a known issue within Spatial4j core. A simple patch in ES will fix the issue.  For now this random test will be disabled until the patch can be applied.
2015-02-26 12:15:55 -06:00
javanna 624549b4eb [TEST] resolved a few compiler warnings in ElasticsearchIntegrationTest
Mainly simplified buildTestCluster method now that there is no global cluster anymore.
2015-02-26 17:04:31 +01:00
Simon Willnauer b3be23ea11 [TEST] filter out unassigned shards 2015-02-26 14:12:27 +01:00
javanna 46461c7730 [TEST] Remove global scope mentions
We removed the global cluster with #9781, yet there were still a few mentions of it in comments. Removed them.
2015-02-26 12:00:30 +01:00
Boaz Leskes de639961a1 Test: fix sync issues in AbstractSimpleTransportTests.testTracerLog 2015-02-26 09:40:36 +01:00
Boaz Leskes 760e738829 Add ClusterHealthResponse constructor needed for plugin BWC 2015-02-26 08:27:35 +01:00
Igor Motov 476b4f1df4 Internal: stop passing default cluster name to cluster state read operations
The default cluster name was needed for backwards compatibility with pre v1.1 nodes. It's no longer needed in 2.0.
2015-02-25 16:44:28 -05:00
Lee Hinman 58cac3d8dd Add a .toString() method to RoutingNode
Previously this would log:

```
[2015-02-24 11:13:45,105][TRACE][cluster.routing.allocation.allocator] [Poltergeist] Try moving shard [[test][2], node[HFn4dJ7fQAyfSAB8BquaSQ], [R], s[STARTED]] from [org.elasticsearch.cluster.routing.RoutingNode@6df2c498]
```

Now it will log:

```
[2015-02-25 11:32:45,182][TRACE][cluster.routing.allocation.allocator] [Solarr] Try moving shard [[my_index][2], node[HDhbU4D9Rx27MSr_72bmWQ], [P], s[STARTED]] from [routingNode ([Solarr][HDhbU4D9Rx27MSr_72bmWQ][Xanadu.domain][192.168.0.4], [5 assigned shards])]
```
2015-02-25 14:04:40 -07:00
Boaz Leskes e9dbfa9ee6 Transport: added a simple request tracer, logging incoming and outgoing requests
The request tracer logs in TRACE level under the `transport.tracer` log and is dynamically configurable with include and exclude arrays to filter out unneeded info. By default all requests are logged with the exception of fault detection pings (fired every second).

add the notion of tracers in the MockTransportService for testing purposes

Closes #9286
2015-02-25 21:33:57 +01:00
Robert Muir b7e49f11ed fix comment 2015-02-25 15:32:45 -05:00
Ryan Ernst 32e042f1c4 Mappings: Lock down _index field
see #8143
closes #9870
2015-02-25 12:24:55 -08:00
Boaz Leskes cc2b00c443 Test: cluster.health rest test should use 0 replicas
Some client test harnesses do not limit the number of replicas based on the number of nodes.
2015-02-25 20:35:39 +01:00
Lee Hinman 2e9ea4abaf Add support for `minimum_should_match` to `simple_query_string`
This behaves similar to the way that `minimum_should_match` works for
the `match` query (in fact it is implemented in the exact same way)

Fixes #6449
2015-02-25 11:35:33 -07:00
Christoph Büscher b16fb69315 DateMath: Fix using time zone when rounding.
Currently rounding in DateMathParser This always done in UTC, even
when another time zone is specified. This is fixed by passing the time zone
down to the rounding logic when it is specified.

Closes #9814
Closes #9885
2015-02-25 19:01:28 +01:00
Boaz Leskes 3e32dd985a Recovery: RecoveryState clean up
To support the `_recovery` API, the recovery process keeps track of current progress in a class called RecoveryState. This class currently have some issues, mostly around concurrency (see #6644 ). This PR cleans it up as well as other issues around it:

- Make the Index subsection API cleaner:
- remove redundant information - all calculation is done based on the underlying file map
- clearer definition of what is what: total files, vs reused files (local files that match the source) vs recovered files (copied over). % based progress is reported based on recovered files only.
- cleaned up json response to match other API (sadly this breaks the structure). We now properly report human values for dates and other units.
- Add more robust unit testing
- Detail flag was passed along as state (it's now a ToXContent param)
- State lookup during reporting is now always done via the IndexShard , no more fall backs to many other classes.
- Cleanup APIs around time and move the little computations to the state class as opposed to doing them out of the API

I also improved error messages out of the REST testing infra for things I run into.

Closes #6644
Closes #9811
2015-02-25 17:34:22 +01:00
Robert Muir 99714ee1bd Reduce tests.policy a bit more
Remove some more wildcard permissions and replace with just what we need.

Closes #9868.
2015-02-25 11:22:02 -05:00
Igor Motov c5ebdf11bb Snapshot/Restore: add ability to retrieve currently running snapshots
Together with #8782 it should help in the situations simliar to #8887 by adding an ability to get information about currently running snapshot without accessing the repository itself.

Closes #8887
2015-02-25 11:06:32 -05:00
Boaz Leskes 06417a85e0 Test: cat health api didn't disable headers 2015-02-25 16:17:47 +01:00
Boaz Leskes 6953777c3a API: add pending tasks count to cluster health
The number of current pending tasks is useful to detect and overloaded master. This commit adds it to the cluster health API. The complete list can be retrieved from the dedicated pending tasks API.

It also adds rest tests for the cluster health variants.

Closes #9877
2015-02-25 14:58:44 +01:00
Simon Willnauer 1ed6451229 [CORE] Allow primary promotion on shadow replica without failing the shard
Today we fail the shard if we need to upgrade a replica to a primary on shadow replicas
on shared filesystem. Yet, this commit allows promotion by re-initializing on the master preventing
reallocation of all replicas.
2015-02-25 14:46:38 +01:00
Simon Willnauer 306b7b0f2b [INDICES] Schedule pending delete if index store delete fails
We try to lock all shards when an index is deleted but likely not
succeeding since shards are still active. To ensure that shards
that used to be allocated on that node get cleaned up as well we have
to retry or block on the delete until we get the locks. This is not desirable
since the delete happens on the cluster state processing thread. Instead of blocking
this commit schedules a pending delete for the index just like if we can't delete shards.
2015-02-25 14:31:18 +01:00
Clinton Gormley 5a53ff6f1b Update migrate_2_0.asciidoc
More code formatting in breaking changes
2015-02-25 14:13:25 +01:00
Clinton Gormley e805fe71cc Update migrate_2_0.asciidoc
Code formatting in breaking changes
2015-02-25 14:11:57 +01:00
Clinton Gormley 5146cf6256 Update migrate_2_0.asciidoc
Fixed bad heading levels in breaking changes
2015-02-25 14:10:17 +01:00
Clinton Gormley 0c61ea803d Update migrate_2_0.asciidoc
Fixed bad asciidoc in breaking changes
2015-02-25 14:07:19 +01:00
Colin Goodheart-Smithe 2520dc78ec [DOCS] added a note for the default shard_size value 2015-02-25 11:00:55 +00:00
Boaz Leskes 674fe9ba28 ClusterService: improve update thread shutdown logic. 2015-02-25 08:49:30 +01:00
Simon Willnauer 98ce594ac8 [TEST] write.lock is a valid file on a cleaned index 2015-02-25 08:45:58 +01:00
Ryan Ernst be0cef0c43 Mappings: Lock down _type field
see #8143
closes #9869
2015-02-24 22:37:41 -08:00
Igor Motov a7f8e636ba Tests: make sure that repository is not stored when old index file is created
Otherwise the fs repository metadata that points to non-existing location is stored in the old index cluster state, which causes warnings during OldIndexBackwardsCompatibilityTests.
2015-02-24 19:28:35 -05:00
Ryan Ernst b96bd201c1 Mappings: Lock down _id field
There are two implications to this change.
First, percolator now uses _uid internally, extracting the id portion
when needed. Second, sorting on _id is no longer possible, since you
can no longer index _id. However, _uid can still be used to sort, and
is better anyways as indexing _id just to make it available to
fielddata for sorting is wasteful.

see #8143
closes #9842
2015-02-24 14:26:22 -08:00