Commit Graph

14503 Commits

Author SHA1 Message Date
Christoph Büscher 7f88cc596b Query refactoring: SpanNotQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new doToQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #12365
2015-07-22 14:50:21 +02:00
Christoph Büscher 22dcc77062 Merge branch 'master' into feature/query-refactoring 2015-07-22 10:38:56 +02:00
Christoph Büscher 1a09633a62 Merge pull request #12182 from cbuescher/feature/query-refactoring-spanmultiterm
Query refactoring: SpanMultiTermQueryBuilder and Parser
2015-07-22 10:37:34 +02:00
Christoph Büscher 966c02b1ac Query refactoring: SpanMultiTermQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #12182
2015-07-22 10:36:15 +02:00
Areek Zillur 01ea89acbf [TEST] Verify expected number of nodes in cluster before issuing shardStores request 2015-07-21 19:30:26 -04:00
Lee Hinman c286cd11cf Ignore EngineClosedException during translog fysnc
When performing an operation on a primary, the state is captured and the
operation is performed on the primary shard. The original request is
then modified to increment the version of the operation as preparation
for it to be sent to the replicas.

If the request first fails on the primary during the translog sync
(because the Engine is already closed due to shadow primaries closing
the engine on relocation), then the operation is retried on the new primary
after being modified for the replica shards. It will then fail due to the
version being incorrect (the document does not yet exist but the request
expects a version of "1").

Order of operations:

- Request is executed against primary
- Request is modified (version incremented) so it can be sent to replicas
- Engine's translog is fsync'd if necessary (failing, and throwing an exception)
- Modified request is retried against new primary

This change ignores the exception where the engine is already closed
when syncing the translog (similar to how we ignore exceptions when
refreshing the shard if the ?refresh=true flag is used).
2015-07-21 15:09:40 -06:00
Jason Tedor 96c245c190 Merge pull request #12380 from jasontedor/fix/elasticsearch-cloud-azure-51
Correctly list blobs in Azure Storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage
2015-07-21 16:47:04 -04:00
Shay Banon be3e315248 Replica allocator unit tests
First batch of unit tests to verify the behavior of replica allocator
2015-07-21 22:37:40 +02:00
Jason Tedor 0b8b147a74 Correctly list blobs in Azure storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage
This commit addresses an issue that was leading to snapshot corruption for snapshots stored as blobs in Azure Storage.

The underlying issue is that in cases when multiple snapshots of an index were taken and persisted into Azure Storage, snapshots subsequent
to the first would repeatedly overwrite the snapshot files. This issue does render useless all snapshots except the final snapshot.

The root cause of this is due to String concatenation involving null. In particular, to list all of the blobs in a snapshot directory in
Azure the code would use the method listBlobsByPrefix where the prefix is null. In the listBlobsByPrefix method, the path keyPath + prefix
is constructed. However, per 5.1.11, 5.4 and 15.18.1 of the Java Language Specification, the reference null is first converted to the string
"null" before performing the concatenation. This leads to no blobs being returned and therefore the snapshot mechanism would operate as if
it were writing the first snapshot of the index. The fix is simply to check if prefix is null and handle the concatenation accordingly.

Upon fixing this issue so that subsequent snapshots would no longer overwrite earlier snapshots, it was discovered that the snapshot metadata
returned by the listBlobsByPrefix method was not sufficient for the snapshot layer to detect whether or not the Lucene segments had already
been copied to the Azure storage layer in an earlier snapshot. This led the snapshot layer to unnecessarily duplicate these Lucene segments
in Azure Storage.

The root cause of this is due to known behavior in the CloudBlobContainer.getBlockBlobReference method in the Azure API. Namely, this method
does not fetch blob attributes from Azure. As such, the lengths of all the blobs appeared to the snapshot layer to be of length zero and
therefore they would compare as not equal to any new blobs that the snapshot layer is going to persist. To remediate this, the method
CloudBlockBlob.downloadAttributes must be invoked. This will fetch the attributes from Azure Storage so that a proper comparison of the
blobs can be performed.

Closes elastic/elasticsearch-cloud-azure#51, closes elastic/elasticsearch-cloud-azure#99
2015-07-21 16:33:33 -04:00
Lee Hinman b230927dbc Unit tests for `nodesAndVersions` on shared filesystems
With the `recover_on_any_node` setting, these unit tests check that the
correct node list and versions are returned.
2015-07-21 14:17:51 -06:00
Robert Muir 14f8671a48 Merge pull request #12381 from rmuir/fail_if_no_tests
don't fail junit4 integration tests if there are no tests.
2015-07-21 14:49:50 -04:00
Robert Muir b91b0eb3e8 don't fail junit4 integration tests if there are no tests.
instead fail the failsafe plugin, which means the external cluster will still get shut down
2015-07-21 14:15:59 -04:00
Clinton Gormley aeb0ca0aef Merge pull request #12377 from clintongormley/license_check_all_packages
Refactored check_license_and_sha.pl to accept a license dir and package path
2015-07-21 19:54:21 +02:00
Alexander Reelsen 988535eb0d Testing: Fix help displaying tests under windows
The help files are using a unix based file separator, where as
the test relies on the help being based on the file system separator.

This commit fixes the test to remove all `\r` characters before
comparing strings.

The test has also been moved into its own CliToolTestCase, as it does
not need to be an integration test.
2015-07-21 18:53:55 +02:00
Clinton Gormley f84757e8bb Refactored check_license_and_sha.pl to accept a license dir and package path
In preparation for the move to building the core zip, tar.gz, rpm, and deb as separate modules, refactored check_license_and_sha.pl to:

* accept a license dir and path to the package to check on the command line
* to be able to extract zip, tar.gz, deb, and rpm
* all packages except rpm will work on Windows
2015-07-21 18:04:52 +02:00
Chris Earle 16418b34a2 Updating breaking changes
- field names cannot be mapped with `.` in them
- fixed asciidoc issue where the list was not recognized as a list
2015-07-21 08:35:28 -07:00
Shay Banon cd7096dfc2 Replace primaryPostAllocated flag and use UnassignedInfo
There is no need to maintain additional state as to if a primary was allocated post api creation on the index routing table, we hold all this information already in the UnassignedInfo class.
closes #12374
2015-07-21 17:00:07 +02:00
Alexander Reelsen d3e454780f PluginManager: Fix bin/plugin calls in scripts/bats test
The release and smoke test python scripts used to install
plugins in the old fashion.

Also the BATS testing suite installed/removed plugins in that
way. Here the marvel tests have been removed, as marvel currently
does not work with the master branch.

In addition documentation has been updated as well, where it was
still missing.
2015-07-21 16:17:59 +02:00
Boaz Leskes 5def4193e9 Discovery: make sure NodeJoinController.ElectionCallback is always called from the update cluster state thread
This is important for correct handling of the joining thread. This causes assertions to trip in our test runs. See http://build-us-00.elastic.co/job/es_g1gc_master_metal/11653/ as an example

Closes #12372
2015-07-21 16:11:37 +02:00
Boaz Leskes 7adf9929ba Remove left over no commit from TransportReplicationAction
It asks to double check thread pool rejection. I did and don't see problems with it.
2015-07-21 15:54:36 +02:00
Alexander Reelsen 999d5ab801 CliTool: Various PluginManager fixes
The new plugin manager parser was not called correctly in the scripts.
In addition the plugin manager now creates a plugins/ directory in case
it does not exist.

Also the integration tests called the plugin manager in the deprecated way.
2015-07-21 15:31:57 +02:00
Alexander Reelsen 2f54b89a23 CLITool: Port PluginManager to use CLITool
In order to unify the handling and reuse the CLITool infrastructure
the plugin manager should make use of this as well.

This obsolets the -i and --install options but requires the user
to use `install` as the first argument of the CLI.

This is basically just a port of the existing functionality, which
is also the reason why this is not a refactoring of the plugin manager,
which will come in a separate commit.
2015-07-21 14:15:39 +02:00
Martijn van Groningen 0cf9c3268c Remove custom execute local logic in TransportSingleShardAction and TransportInstanceSingleOperationAction and rely on transport service to execute locally. (forking thread etc.)
Change TransportInstanceSingleOperationAction to have shardActionHandler to, so we can execute locally without endless spinning.
2015-07-21 13:45:42 +02:00
Alexander Reelsen 95f56d71bd Logging: Corrected potential confusing log message 2015-07-21 11:30:26 +02:00
Christoph Büscher 37cdc1344a Merge branch 'master' into feature/query-refactoring 2015-07-21 10:58:58 +02:00
Ryan Ernst e52a316609 Test: fix tests referring to _index 2015-07-21 01:24:49 -07:00
Ryan Ernst ededbf0c8b Merge pull request #12352 from rjernst/fix/index-name-in-mapper-service
Remove index name from mapping parser
2015-07-21 00:31:38 -07:00
Ryan Ernst 8f6398a8f7 Merge pull request #12356 from rjernst/fix/12329
Remove ability to configure _index
2015-07-21 00:30:00 -07:00
Ryan Ernst 8cd03cce5e Merge branch 'master' into fix/12329 2015-07-21 00:29:34 -07:00
Ryan Ernst f4ba5ac6b8 Merge pull request #12357 from rjernst/fix/12317
Update time_zone specification
2015-07-21 00:27:27 -07:00
Ryan Ernst dba42a83e2 Docs: Update time_zone specification
closes #12317
2015-07-21 00:22:53 -07:00
Boaz Leskes bb1318512b Test: move IndicesShardStoreRequestTests to the right package 2015-07-21 09:01:50 +02:00
Boaz Leskes 29c99ff23d Logging: shard store shouldn't log the entire cluster state under trace, but just its version.
If we want the entire CS, we can put the cluster.service on trace
2015-07-21 08:58:50 +02:00
Ryan Ernst 1c99626b84 Mappings: Remove ability to configure _index
The `_index` field is now a completely virtual field thanks
to #12027. It is no longer necessary to index the actual value
of the index name.

closes #12329
2015-07-20 23:54:35 -07:00
Ryan Ernst abcf269539 Remove old DocumentMapper.parse method 2015-07-20 22:37:44 -07:00
Areek Zillur ae750bca69 [TEST] add missing ensureYellow to IndicesShardStoreRequestTests.testBasic 2015-07-20 23:52:07 -04:00
Ryan Ernst 2ad682e8b4 Mappings: Remove index name from mapping parser
The index name was passed along through many levels of mapping parsing,
just so that it could be used for _index. However, the index name
is really metadata that should exist alongside things like type and
id in SourceToParse.

This change moves index name to SourceToParse, and eliminates it from the
DocumentMapperParser.
2015-07-20 17:50:07 -07:00
Martijn van Groningen 4e2bdb67b7 test: enable get field mapping tests 2015-07-20 23:38:47 +02:00
Areek Zillur e964b0c2ab [TEST] use official API to read cluster state and enable debug logging 2015-07-20 17:12:30 -04:00
Boaz Leskes 66e71da7bd Test: testTimeoutCleanup use assertBusy while checking timer queue 2015-07-20 22:38:53 +02:00
Jason Tedor 34155ff102 Merge pull request #12346 from jasontedor/fix/12345
Use time with nanosecond resolution calculated at the executing node to measure the time that contexts are held open
2015-07-20 15:54:54 -04:00
Boaz Leskes 068fae218f Test: testShardActiveElseWhere shouldn't force cluster state publishing
The test is supposed to fool a node, making it think all it's shards are allocated else where. The test verifies the node doesn't delete its shards. The current test submitted this "fake" CS to the master, causing it to published which in turn caused confusion on other nodes and test failures.

 See http://build-us-00.elastic.co/job/es_core_master_oracle_6/990/testReport/junit/org.elasticsearch.indices.store/IndicesStoreIntegrationTests/testShardActiveElseWhere/
2015-07-20 21:16:16 +02:00
Zachary Tong 9951db9c64 [TEST] Better floating point comparisons 2015-07-20 14:46:24 -04:00
Jason Tedor 3af763eb88 Use time with nanosecond resolution calculated at the executing node to measure the time that contexts are held open
Closes #12345
2015-07-20 12:42:13 -04:00
Martijn van Groningen 4c7bb17521 test: increased logging 2015-07-20 17:35:39 +02:00
Robert Muir cd9d1c617f Merge pull request #12344 from rmuir/2.1.16
Update randomizedtesting to 2.1.16
2015-07-20 11:13:57 -04:00
Robert Muir a0a6299840 Update randomizedtesting to 2.1.16 2015-07-20 11:02:50 -04:00
Shay Banon 32445bbc3a Simplify handling of ignored unassigned shards
Fold ignored unassigned to a UnassignedShards and have simpler handling of them. Also remove the trapy way of adding an ignored unassigned shards today directly to the list, and have dedicated methods for it.

This change also removes the useless moving of unassigned shards to the end, since anyhow we first, sort those unassigned shards, and second, we now have persistent "store exceptions" that should not cause "dead letter" shard allocation.
2015-07-20 15:43:31 +02:00
Shay Banon ca3e0c6d49 Initial Refactor Gateway Allocator
Break it into more manageable code by separating allocation primaries and allocating replicas. Start adding basic unit tests for primary shard allocator.
2015-07-20 14:04:24 +02:00
Christoph Büscher 7540ad5d50 Merge branch 'master' into feature/query-refactoring 2015-07-20 12:52:32 +02:00