Commit Graph

11528 Commits

Author SHA1 Message Date
Ryan Ernst 9d708e20a0 Mappings: Lock down _size field
This also changes the stored setting for _size to true (for
indexes created in 2.x).

see #8143
closes #9913
2015-02-27 11:09:52 -08:00
Ryan Ernst f87fa8172e Tests: Add back tests.cluster
This setting is used by the release script to run rest tests against
the version being released.  It used to work only for tests using
the global cluster.  Now it supercedes both SUITE and TEST scope
test clusters.

closes #9916
2015-02-27 11:08:24 -08:00
Robert Muir 4ca57ab10a Upgrade to Lucene r1662607
Closes #9915.

Squashed commit of the following:

commit cfa59f5a3f03d9d1b432980dcee6495447c1e7ea
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Feb 27 12:10:16 2015 -0500

    add missing null check

commit 62fe5403068c730c0e0b6fd1ab1a0246eeef6220
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Feb 27 11:31:53 2015 -0500

    Disable ExtrasFS for now, until we hook all this in properly in a separate issue.

commit 822795c57c5cf846423fad443c2327c4ed0094ac
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Feb 27 10:12:02 2015 +0100

    Fix PercolatorTests.

commit 98b2a0a7d8298648125c9a367cb7e31b3ec7d51b
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Feb 27 09:27:11 2015 +0100

    Fix ChildrenQueryTests.

commit 9b99656fc56bbd01c9afe22baffae3c37bb48a71
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Feb 26 20:50:02 2015 -0500

    cutover apis, no work on test failures yet.
2015-02-27 12:51:36 -05:00
jaymode efbda318d0 Tests: check node ports availability when using unicast discovery
Some tests failures are seen when a node attempts to use a port that is already bound
by some other process on the test machine. This commit adds a bind to test port availability
and iterates over the port range until an available port is found. This reduces the likelihood
of a test node failing to start up due to the port already being bound.
2015-02-27 11:49:54 -05:00
Simon Willnauer ce53e20351 [CORE] Remove Restart API and remove Node#stop()
The restart API seems pretty esoteric and is entirely untested.
This commit removes it without a replacement.

Closes #9841
2015-02-27 13:58:23 +01:00
Simon Willnauer 2b8827d75c [CORE] Remove component settings from AbstractComponent
Today we have two ways of getting a setting, either with the full settings key or with only
the last part of the key where the prefix is implicit depending on the package the class is in via
component settings. this is trappy as well as confusing for users and can break easily if a class is moved
to a new package since the prefix then implicitly changes.
This commit removes the component settings from the codebase.
2015-02-27 13:01:26 +01:00
Ryan Ernst e221dc20a4 remove thread leak filter, should have been removed in #9843 2015-02-26 23:24:56 -08:00
Ryan Ernst 3b7928d568 Mappings: Lock down _field_names field
Now that we have an explicit `enabled` flag, we can lock down
the field type so it is not mungeable.

see #8143
closes #9912
2015-02-26 15:15:59 -08:00
Ryan Ernst 7181bbde26 Mappings: Remove _boost field
This has been deprecated since 1.0.0.RC1. It is finally removed here.

closes #8875
2015-02-26 15:07:07 -08:00
Ryan Ernst cc8f159a3c fix bwc test to be more lenient on error message when _field_names is
not supported
2015-02-26 14:32:50 -08:00
Ryan Ernst 78df69e6a0 Mappings: Lock down _routing field
`required` is now the only changeable settings (on indexes created after 1.x).

see #8143
closes #9895
2015-02-26 13:09:41 -08:00
Ryan Ernst 257cdcd110 Mappings: Add enabled flag for _field_names to replace disabling through index=no
Almost all of our meta fields that allow enabling/disabling have an `enabled`
setting.  However, _field_names is enabled by default, and disabling
requires setting `index=no`.  This change adds a flag similar to that
with other meta fields.

closes #9893
2015-02-26 13:01:07 -08:00
Simon Willnauer 261e21a386 [TEST] Fix NPE in ElasticsearchIntegrationTest if no indexer is provided
Closes #9907
2015-02-26 21:13:55 +01:00
Nicholas Knize b05f5ebee8 [TEST] Adding 'AwaitsFix' annotation to randomGeoCollection test
Random geo shape testing periodically fails on a known issue within Spatial4j core. A simple patch in ES will fix the issue.  For now this random test will be disabled until the patch can be applied.
2015-02-26 12:15:55 -06:00
javanna 624549b4eb [TEST] resolved a few compiler warnings in ElasticsearchIntegrationTest
Mainly simplified buildTestCluster method now that there is no global cluster anymore.
2015-02-26 17:04:31 +01:00
Simon Willnauer b3be23ea11 [TEST] filter out unassigned shards 2015-02-26 14:12:27 +01:00
javanna 46461c7730 [TEST] Remove global scope mentions
We removed the global cluster with #9781, yet there were still a few mentions of it in comments. Removed them.
2015-02-26 12:00:30 +01:00
Boaz Leskes de639961a1 Test: fix sync issues in AbstractSimpleTransportTests.testTracerLog 2015-02-26 09:40:36 +01:00
Boaz Leskes 760e738829 Add ClusterHealthResponse constructor needed for plugin BWC 2015-02-26 08:27:35 +01:00
Igor Motov 476b4f1df4 Internal: stop passing default cluster name to cluster state read operations
The default cluster name was needed for backwards compatibility with pre v1.1 nodes. It's no longer needed in 2.0.
2015-02-25 16:44:28 -05:00
Lee Hinman 58cac3d8dd Add a .toString() method to RoutingNode
Previously this would log:

```
[2015-02-24 11:13:45,105][TRACE][cluster.routing.allocation.allocator] [Poltergeist] Try moving shard [[test][2], node[HFn4dJ7fQAyfSAB8BquaSQ], [R], s[STARTED]] from [org.elasticsearch.cluster.routing.RoutingNode@6df2c498]
```

Now it will log:

```
[2015-02-25 11:32:45,182][TRACE][cluster.routing.allocation.allocator] [Solarr] Try moving shard [[my_index][2], node[HDhbU4D9Rx27MSr_72bmWQ], [P], s[STARTED]] from [routingNode ([Solarr][HDhbU4D9Rx27MSr_72bmWQ][Xanadu.domain][192.168.0.4], [5 assigned shards])]
```
2015-02-25 14:04:40 -07:00
Boaz Leskes e9dbfa9ee6 Transport: added a simple request tracer, logging incoming and outgoing requests
The request tracer logs in TRACE level under the `transport.tracer` log and is dynamically configurable with include and exclude arrays to filter out unneeded info. By default all requests are logged with the exception of fault detection pings (fired every second).

add the notion of tracers in the MockTransportService for testing purposes

Closes #9286
2015-02-25 21:33:57 +01:00
Robert Muir b7e49f11ed fix comment 2015-02-25 15:32:45 -05:00
Ryan Ernst 32e042f1c4 Mappings: Lock down _index field
see #8143
closes #9870
2015-02-25 12:24:55 -08:00
Boaz Leskes cc2b00c443 Test: cluster.health rest test should use 0 replicas
Some client test harnesses do not limit the number of replicas based on the number of nodes.
2015-02-25 20:35:39 +01:00
Lee Hinman 2e9ea4abaf Add support for `minimum_should_match` to `simple_query_string`
This behaves similar to the way that `minimum_should_match` works for
the `match` query (in fact it is implemented in the exact same way)

Fixes #6449
2015-02-25 11:35:33 -07:00
Christoph Büscher b16fb69315 DateMath: Fix using time zone when rounding.
Currently rounding in DateMathParser This always done in UTC, even
when another time zone is specified. This is fixed by passing the time zone
down to the rounding logic when it is specified.

Closes #9814
Closes #9885
2015-02-25 19:01:28 +01:00
Boaz Leskes 3e32dd985a Recovery: RecoveryState clean up
To support the `_recovery` API, the recovery process keeps track of current progress in a class called RecoveryState. This class currently have some issues, mostly around concurrency (see #6644 ). This PR cleans it up as well as other issues around it:

- Make the Index subsection API cleaner:
- remove redundant information - all calculation is done based on the underlying file map
- clearer definition of what is what: total files, vs reused files (local files that match the source) vs recovered files (copied over). % based progress is reported based on recovered files only.
- cleaned up json response to match other API (sadly this breaks the structure). We now properly report human values for dates and other units.
- Add more robust unit testing
- Detail flag was passed along as state (it's now a ToXContent param)
- State lookup during reporting is now always done via the IndexShard , no more fall backs to many other classes.
- Cleanup APIs around time and move the little computations to the state class as opposed to doing them out of the API

I also improved error messages out of the REST testing infra for things I run into.

Closes #6644
Closes #9811
2015-02-25 17:34:22 +01:00
Robert Muir 99714ee1bd Reduce tests.policy a bit more
Remove some more wildcard permissions and replace with just what we need.

Closes #9868.
2015-02-25 11:22:02 -05:00
Igor Motov c5ebdf11bb Snapshot/Restore: add ability to retrieve currently running snapshots
Together with #8782 it should help in the situations simliar to #8887 by adding an ability to get information about currently running snapshot without accessing the repository itself.

Closes #8887
2015-02-25 11:06:32 -05:00
Boaz Leskes 06417a85e0 Test: cat health api didn't disable headers 2015-02-25 16:17:47 +01:00
Boaz Leskes 6953777c3a API: add pending tasks count to cluster health
The number of current pending tasks is useful to detect and overloaded master. This commit adds it to the cluster health API. The complete list can be retrieved from the dedicated pending tasks API.

It also adds rest tests for the cluster health variants.

Closes #9877
2015-02-25 14:58:44 +01:00
Simon Willnauer 1ed6451229 [CORE] Allow primary promotion on shadow replica without failing the shard
Today we fail the shard if we need to upgrade a replica to a primary on shadow replicas
on shared filesystem. Yet, this commit allows promotion by re-initializing on the master preventing
reallocation of all replicas.
2015-02-25 14:46:38 +01:00
Simon Willnauer 306b7b0f2b [INDICES] Schedule pending delete if index store delete fails
We try to lock all shards when an index is deleted but likely not
succeeding since shards are still active. To ensure that shards
that used to be allocated on that node get cleaned up as well we have
to retry or block on the delete until we get the locks. This is not desirable
since the delete happens on the cluster state processing thread. Instead of blocking
this commit schedules a pending delete for the index just like if we can't delete shards.
2015-02-25 14:31:18 +01:00
Clinton Gormley 5a53ff6f1b Update migrate_2_0.asciidoc
More code formatting in breaking changes
2015-02-25 14:13:25 +01:00
Clinton Gormley e805fe71cc Update migrate_2_0.asciidoc
Code formatting in breaking changes
2015-02-25 14:11:57 +01:00
Clinton Gormley 5146cf6256 Update migrate_2_0.asciidoc
Fixed bad heading levels in breaking changes
2015-02-25 14:10:17 +01:00
Clinton Gormley 0c61ea803d Update migrate_2_0.asciidoc
Fixed bad asciidoc in breaking changes
2015-02-25 14:07:19 +01:00
Colin Goodheart-Smithe 2520dc78ec [DOCS] added a note for the default shard_size value 2015-02-25 11:00:55 +00:00
Boaz Leskes 674fe9ba28 ClusterService: improve update thread shutdown logic. 2015-02-25 08:49:30 +01:00
Simon Willnauer 98ce594ac8 [TEST] write.lock is a valid file on a cleaned index 2015-02-25 08:45:58 +01:00
Ryan Ernst be0cef0c43 Mappings: Lock down _type field
see #8143
closes #9869
2015-02-24 22:37:41 -08:00
Igor Motov a7f8e636ba Tests: make sure that repository is not stored when old index file is created
Otherwise the fs repository metadata that points to non-existing location is stored in the old index cluster state, which causes warnings during OldIndexBackwardsCompatibilityTests.
2015-02-24 19:28:35 -05:00
Ryan Ernst b96bd201c1 Mappings: Lock down _id field
There are two implications to this change.
First, percolator now uses _uid internally, extracting the id portion
when needed. Second, sorting on _id is no longer possible, since you
can no longer index _id. However, _uid can still be used to sort, and
is better anyways as indexing _id just to make it available to
fielddata for sorting is wasteful.

see #8143
closes #9842
2015-02-24 14:26:22 -08:00
Simon Willnauer c54bd2f7ad [STORE] Improve safety when deleting files from the store
Today if we delete files from the index directory we never acquire the
write lock. Yet, for safety reasons we should get the write lock before
we modify / delete any files. Where we can we should leave the deletion
to the index writer and only delete that are necessary to delete ourself.
2015-02-24 21:15:19 +01:00
Igor Motov 73faa2e5ed Snapshot/Restore: Allow deletion of snapshots with corrupted snapshot files
Improve resiliency of snapshot deletion operation by allowing deletion of snapshot with corrupted snapshot files.

Closes #9534
2015-02-24 14:31:49 -05:00
Ryan Ernst d479611dc5 Tests: Use all found index files instead of static list for static bwc tests
It is a pain to add a new static index, and then have to update the old
index test.  This removes the need for the latter step.

closes #9854
2015-02-24 11:19:57 -08:00
Igor Motov 432f578807 Internal: refactor settings filtering
Refactor how settings filters are handled. Instead of specifying settings filters as filtering class, settings filters are now specified as list of settings that needs to be filtered out. Regex syntax is supported. This is breaking change and will require small change in plugins that are using settingsFilters. This change is needed in order to simplify cluster state diff implementation.

Contributes to #6295
2015-02-24 13:05:47 -05:00
Britta Weber ff8fd677cc mappings: update cluster state with type mapping also for failed indexing request
When indexing of a document with a type that is not in the mappings fails,
for example because "dynamic": "strict" but doc contains a new field,
then the type is still created on the node that executed the indexing request.
However, the change was never added to the cluster state.
This commit makes sure mapping updates are always added to the cluster state
even if indexing of a document fails.

closes #8692
relates to #8650
2015-02-24 17:41:12 +01:00
Ryan Ernst 4ee7ed987e Tests: Remove thread leak filter
Now that the global cluster is gone, we shoudln't need to ignore
thread leaks across tests.  We unfortunately still need suite level
scope, since most tests are using suite scope clusters (although
test clope clusters should really switch back to test scope thread
leaks).

closes #9843
2015-02-24 08:06:10 -08:00