Commit Graph

815 Commits

Author SHA1 Message Date
Britta Weber 6754a26d08 more generic? 2015-07-29 16:46:13 +02:00
Britta Weber d113b59e86 add SearchContext to innerParse method 2015-07-29 16:06:40 +02:00
Britta Weber 6295738d4c make Context generic so we don't have to cast 2015-07-29 16:00:37 +02:00
Britta Weber 24c4b27978 add some documentation 2015-07-29 15:44:58 +02:00
Britta Weber 3ee3238d54 remove unused imports and vars 2015-07-29 15:44:58 +02:00
Britta Weber ea14f675b2 remove hasFetchSubPhaseContext 2015-07-29 15:44:58 +02:00
Britta Weber df339f4cb9 toArray 2015-07-29 15:44:58 +02:00
Britta Weber 5f0f66908e make fielddata fields a plugin 2015-07-29 15:44:58 +02:00
Britta Weber bcebb61f25 make fetch sub phases pluggable 2015-07-29 15:44:58 +02:00
Alexander Reelsen 86ab892986 Releases: Added version numbers & bwc indices for 1.6.2/1.7.1 2015-07-29 12:31:39 +02:00
Christoph Büscher a1948350a5 Merge branch 'master' into feature/query-refactoring 2015-07-29 10:51:08 +02:00
Martijn van Groningen 5d7ed70fa0 Merge pull request #12518 from martijnvg/top_hits/bug/protected_against_crazy_size
Protected against  `size` and `offset` larger than total number of document in a shard
2015-07-29 10:33:27 +02:00
Dave Parfitt dc597f0bf5 Merge pull request #12509 from metadave/dp_snake_case_dates
Default date formats to use underscores via PUT
2015-07-28 16:57:45 -04:00
Lee Hinman 2dc4550839 Add more debugging information to the Awareness Decider
Relates to #12431
2015-07-28 13:48:40 -06:00
Jason Tedor 37177d2f4c Use consistent check whether or not timeout is set 2015-07-28 15:44:55 -04:00
Martijn van Groningen 3f6e470810 top_hits: If topN (based on `offset` + `size`) is higher than the maxDoc of an shard then normalize topN to maxDoc.
Closes #12510
2015-07-28 21:38:58 +02:00
Nik Everett d8a6ddf580 Merge _all: Stop NPE querying _all when it doesn't exist 2015-07-28 10:18:17 -04:00
Dave Parfitt af17b9cd40 Use underscores for date formats
@clintongormley

Closes #12429
2015-07-28 10:04:28 -04:00
Nik Everett adcd1fc11d _all: Add missing boost
When we rewrite to a MatchNoTermsQuery we were throwing out the boost which
could could lead to funky changes when the query against _all was in a
bool query.
2015-07-28 09:25:06 -04:00
Colin Goodheart-Smithe 2ebf229632 Aggregations: Fixes serialization of HDRHistogram in percentiles aggregations
Previously we would write the entire ByteBuffer to the stream to serialise the HDRHistogram even if it was not all needed. Now we only write the bytes that are actually written to in the ByteBuffer.
2015-07-28 14:07:06 +01:00
javanna f6ca11db30 [TEST] remove createExpectedQuery from BaseQueryTestCase
We currently test the toQuery method by comparing its return value with the output of createExpectedQuery. This seemed at first like a good deep test for the lucene generated queries, but ends up causing a lot of code duplication between tests and prod code, which smells like this test is not that useful after all.

This commit removes the requirement for createExpectedQuery, we test a bit less the lucene generated queries, but we do still have a testToQuery method in BaseQueryTestCase. This test acts as smoke test to verify that there are no issues with our toQuery method implementation for each query: it verifies that two equal queries should result in the same lucene query. It also verifies that changing the boost value of a query affects the result of the toQuery.

Part of the previous createExpectedQuery can be moved to assertLuceneQuery using normal assertions, we do it when it makes sense only though.

This commit also adds support for boost and queryName to SpanMultiTermQueryParser and SpanMultiTermQueryBuilder, plus it removes no-op setters for queryName and boost from QueryFilterBuilder. Instead we simply declare in our test infra that query filter doesn't support boost and queryName and we disable tests aroudn these fields in that specific case. Boost and queryName support is also moved down to the QueryBuilder interface.

Closes #12473
2015-07-28 14:24:50 +02:00
Christoph Büscher b780346cb7 Merge branch 'master' into feature/query-refactoring 2015-07-28 11:28:45 +02:00
Colin Goodheart-Smithe 7ab9bbbfad [Test] Muted all HDRPercentilesTests and HDRPercentileRanksTests 2015-07-28 10:17:47 +01:00
Nik Everett d7491515b2 _all: Stop NPE querying _all when it doesn't exist
This can happen in two ways:
1. The _all field is disabled.
2. There are documents in the index, the _all field is enabled, but there are
no fields in any of the documents.

In both of these cases we now rewrite the query to a MatchNoDocsQuery which
should be safe because there isn't anything to match.

Closes #12439
2015-07-27 16:23:32 -04:00
Boaz Leskes 616941e902 Internal: IndicesStore shouldn't try to delete index after deleting a shard
When a node discovers shard content on disk which isn't used, we reach out to all other nodes that supposed to have the shard active. Only once all of those have confirmed the shard active, the shard has no unassigned copies *and* no cluster state change have happened in the mean while, do we go and delete the shard folder.

Currently, after removing a shard, the IndicesStores checks the indices services if that has no more shard active for this index and if so, it tries to delete the entire index folder (unless on master node, where we keep the index metadata around). This is wrong as both the check and the protections in IndicesServices.deleteIndexStore make sure that there isn't any shard *in use* from that index. However, it may be the we erroneously delete other unused shard copies on disk, without the proper safety guards described above.

Normally, this is not a problem as the missing copy will be recovered from another shard copy on another node (although a shame). However, in extremely rare cases involving multiple node failures/restarts where all shard copies are not available (i.e., shard is red) there are race conditions which can cause all shard copies to be deleted.

Instead, we should change the decision to clean up an index folder to based on checking the index directory for being empty and containing no shards.
2015-07-27 16:09:25 -04:00
Lee Hinman b8b96679ea [TEST] Mark 2 HDRPercentileRanks tests with AwaitsFix 2015-07-27 11:40:20 -06:00
Jason Tedor cdd2da1e1c Check for existence of snapshot index file before attempting delete 2015-07-27 13:28:37 -04:00
Alexander Reelsen 9628d2632f Build: Split packages into submodules
This change creates a proper `distribution` modules in which we have today packaging for
all of our four current packages:

* zip
* tar.gz
* rpm
* deb

Licenes have moved into the distribution project as well. So have the config/ and the bin/ directory
from the core/ project.

The RPM package is now built, if rpmbuild exists.

The bats tests have been moved as well.

Also the zip distribution now executes the REST integration tests.
2015-07-27 17:50:54 +02:00
Tanguy Leroux 01ae730023 Remove Core Lib directory 2015-07-27 16:52:10 +02:00
Lee Hinman 5fdc6cfc24 Revert "Revert "Consistently name Groovy scripts with the same content""
This reverts commit a33cfe4b11.
2015-07-27 08:49:47 -06:00
Colin Goodheart-Smithe 26a299d0e7 [TEST] muted HDRPercentilesTests.testScript_ExplicitSingleValued_WithParams 2015-07-27 15:42:47 +01:00
Adrien Grand a832987ada Forbid Files.isHidden.
As Robert pointed out on #12465, it has the undesirable property of relying on
the operating system. So it would be better to use a simple rule such as
checking whether the file name starts with a dot.
2015-07-27 16:13:04 +02:00
Christoph Büscher 0128252075 Merge branch 'master' into feature/query-refactoring
core/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java
2015-07-27 14:16:39 +02:00
Martijn van Groningen a14913f7b6 Left over from the `query_cache` to `request_cache` rename. 2015-07-27 13:28:15 +02:00
Britta Weber 20facccaf6 Merge pull request #11666 from brwe/meta-data-zen
[TEST] remove redundant tests and move to different suite
2015-07-27 12:40:11 +02:00
Britta Weber e44c5ff703 check if index is closed or was previously closed when gathering relevant indices to write meta state
When an index is opened it will not be assigned to a node but also not have closed state
anymore. Before we only checked if an index either is closed or assigned to the data node
and therefore the change from close->open was not written.
2015-07-27 12:35:25 +02:00
Britta Weber 2713e903ab [TEST] remove redundant tests and move to different suite
Some of the test for meta data are redundant. Also, since they
somewhat test service disruptions (start master with empty
data folder) we might move them to DiscoveryWithServiceDisruptionsTests.
Also, this commit adds a test for
https://github.com/elastic/elasticsearch/issues/11665
2015-07-27 12:35:13 +02:00
Adrien Grand ec79568e6f Merge pull request #12430 from ruflin/ip_range-mask-fix
Fix cidr mask conversion issue for 0.0.0.0/0 and add tests #12005
2015-07-27 12:12:20 +02:00
javanna 8e931d5767 Query DSL: don't cache type filter in DocumentMapper
If we cache the type filter and we e.g. set its boost which is now settable on all queries, the boost will change for all subsequent queries. We should rather create a new query every time.
2015-07-27 11:35:19 +02:00
Ruflin 91d1789258 Move index creation to test setup method 2015-07-27 10:39:04 +02:00
Adrien Grand 00ae4bb792 Merge pull request #12465 from xuzha/xu-skip-hidden
skip hidden files
2015-07-27 10:06:45 +02:00
Ruflin b8c2f05ff5 Revert change to set longTo to "MAX_IP -1" and improve test suite to check for range 2015-07-27 09:11:04 +02:00
Shay Banon a99ccb6112 No need to find replica copy when index is created
There is no need to try and go fetch replica copies for best allocation when the index is created
2015-07-27 07:01:48 +09:00
xuzha b0b9c121c8 skip hidden files 2015-07-26 01:16:29 -07:00
Jason Tedor b3272fe648 Preserve the root cause when encountering an exception expecting to have reached the end of the setting stream 2015-07-25 15:29:15 -04:00
Martijn van Groningen c73fff799d Cleaned up the data structures used in MetaData for alias and index lookups.
Major changes:
* Changed MetaData to holds alias and index lookup information into a single TreeMap instead of two separate maps.
* Moved the building of the alias / index lookup to the metadata builder.
2015-07-24 22:04:11 +02:00
Jason Tedor 267afe866f Merge pull request #12451 from jasontedor/fix/12382
Add explicit check that we have reached the end of the settings stream when parsing settings
2015-07-24 13:52:00 -04:00
Christoph Büscher 3d53413650 Merge branch 'master' into feature/query-refactoring 2015-07-24 19:19:37 +02:00
Jason Tedor 8ac105dbb9 Add explicit check that we have reached the end of the settings stream when parsing settings
Settings are currently parsed by looping over the tokens until an END_OBJECT token is reached. However, this does not mean that the end of
the settings stream was reached. This can occur, for example, when parsing a YAML settings file with inconsistent indentation. Currently
in this case, some settings will be silently ignored. This commit forces a check that we have in fact reached the end of the settings
stream.

Closes #12382
2015-07-24 13:12:44 -04:00
Colin Goodheart-Smithe 3e0532a0c5 Aggregations: Add HDRHistogram as an option in percentiles and percentile_ranks aggregations
HDRHistogram has been added as an option in the percentiles and percentile_ranks aggregation. It has one option `number_significant_digits` which controls the accuracy and memory size for the algorithm

Closes #8324
2015-07-24 17:55:36 +01:00
javanna dbec71face Query DSL: assign result of validateInnerQuery in ConstantScoreQueryBuilder 2015-07-24 18:52:48 +02:00
javanna cf570ac5bd Query DSL: don't cache type filter in DocumentMapper
If we cache the type filter and we e.g. set its boost which is now settable on all queries, the boost will change for all subsequent queries. We should rather create a new query every time.
2015-07-24 18:37:17 +02:00
Christoph Büscher db5d73a617 Merge pull request #12444 from cbuescher/feature/query-refactoring-nullCheckCleanup
Query Refactoring: Move null-checks to validate
2015-07-24 18:17:36 +02:00
Adrien Grand e62aaa928e Tests: Fix TermVectorsUnitTests to not rely on HashMap iteration order. 2015-07-24 17:59:36 +02:00
Christoph Büscher 081d264f40 Merge branch 'master' into feature/query-refactoring 2015-07-24 17:57:17 +02:00
Christoph Büscher 15a09a04f2 Query Refactoring: Move null-checks from constructors and setters to validate
Following up to #12427, this PR does same changes, moving null-checks from construtors
and setters in query builder to the validate() method.

PR against query-refactoring branch
2015-07-24 17:36:25 +02:00
Robert Muir 980e564caa Merge branch 'master' into plugin_java_version 2015-07-24 11:12:52 -04:00
Alex Ksikes 76c03df035 Make sure filter is correctly parsed for multi-term vectors
This makes sure the `filter` parameter is correctly parsed in a multi-term
vector request when using `ids` and `parameters`.

Closes #12311
Closes #12312
2015-07-24 17:09:16 +02:00
Boaz Leskes 57cbce2824 Allocation: ThrottlingAllocationDecider should not counting relocating shards
The ThrottlingAllocationDecider is responsible to limit the number of incoming/local recoveries on a node. It therefore shouldn't count shards marked as relocating which represent the source of the recovery.

Closes #12409
2015-07-24 16:29:51 +02:00
Martijn van Groningen e997342da4 Added IndexSearcherWrapper extension point.
This extension point allows one IndexSearcherWrapper instance to intercept the searcher from the Engine before it is used for a opertion.
2015-07-24 15:52:18 +02:00
Christoph Büscher 8776463084 Merge pull request #12440 from cbuescher/feature/query-refactoring-constructorCleanup
Extended SpanFirstQueryBuilderTest
2015-07-24 15:45:39 +02:00
Christoph Büscher 1c3e8e27bf Extended SpanFirstQueryBuilderTest 2015-07-24 15:19:20 +02:00
Alex Ksikes 94d13c73c7 Refactors WildcardQueryBuilder and Parser
Relates to #10217
Closes #12110

This PR is against the query-refactoring branch.
2015-07-24 14:46:49 +02:00
Christoph Büscher 52113e7527 Merge pull request #12417 from cbuescher/remove-rangequery-context
Remove unused QueryParseContext argument in MappedFieldType#rangeQuery()
2015-07-24 14:19:53 +02:00
Adrien Grand 13d790bd4c Merge pull request #12423 from khiraiwa/patch-1
Interval time is different from the doc value.
2015-07-24 14:19:32 +02:00
Christoph Büscher 817b992511 Merge pull request #12427 from cbuescher/feature/query-refactoring-constructorCleanup
Moving null-checks from constructors to validate()
2015-07-24 14:16:54 +02:00
Martijn van Groningen c781c89e2c ShardUtils#getElasticsearchLeafReader() should use FilterLeafReader#getDelegate() instead of FilterLeafReader#unwrap
If there are multiple levels of filtered leaf readers then with the unwrap() method it immediately returns the most inner leaf reader and thus skipping of over any other filtered leaf reader that may be instance of ElasticsearchLeafReader. By using #getDelegate() method we can check each filter reader layer if it is instance of ElasticsearchLeafReader, so that we never skip over any wrapped filtered leaf reader and lose the shard id.
2015-07-24 13:49:07 +02:00
Boaz Leskes 316d1cb6c0 Core: Adapt IndicesClusterStateService to use allocation ids
Also, as set of utility methods was introduced on ShardRouting to do various types of matching with other shard routings, giving control about what exactly should be matched (same shard id, same allocation id, all but version and shard info etc.). This is useful here, but also prepares the grounds for the change needed in #12387 (making ShardRouting.equals be strict and perform exact equality).

Closes #12397
2015-07-24 13:09:47 +02:00
Alex Ksikes 7099e6ca93 Refactors ScriptQueryBuilder and Parser
Relates to #10217
Closes #12115

This PR is against the query-refactoring branch.
2015-07-24 12:16:44 +02:00
Shay Banon fc90c2affd Cancel replica recovery on another sync option copy found
When a replica is initializing from the primary, and we find a better node that has full sync id match, it is better to cancel the existing replica allocation and allocate it to the new node with sync id match (eventually)
2015-07-24 12:13:03 +02:00
Alex Ksikes 4d4dc5cceb Refactors TypeQuery
Relates to #10217
Closes #12035

This PR is against the query-refactoring branch.
2015-07-24 12:08:48 +02:00
Alex Ksikes 37c6347bce Refactors PrefixQuery
Relates to #10217
Closes #12032

This PR is against the query-refactoring branch.
2015-07-24 11:46:55 +02:00
Christoph Büscher 4df452a915 Moving null-checks from constructors to validate() 2015-07-24 11:13:39 +02:00
Alex Ksikes a4e9f642c2 Refactoring of MissingQuery
Relates to #10217
Closes #12030

This PR is against the query-refactoring branch.
2015-07-24 10:08:28 +02:00
Jay Modi f8d9de8dca Merge pull request #12419 from jaymode/fix_settings_classloader
Copy the classloader from the original settings when checking for prompts
2015-07-23 13:06:09 -04:00
jaymode 8472775477 copy the classloader from the original settings when checking for prompts
Today, when a user provides settings and specifies a classloader to be used, the classloader gets
dropped when we copy the settings to check for prompt entries. This change copies the classloader
when replacing the prompt placeholders and adds a test to ensure the InternalSettingsPreparer
always retains the classloader.

Closes #12340
2015-07-23 12:53:27 -04:00
Alex Ksikes 93674b8b1b Refactoring of RegexpQuery
Relates to #10217
Closes #11896

This PR is against the query-refactoring branch.
2015-07-23 17:40:53 +02:00
Jason Tedor 878301c795 Dump stack traces of all threads on test failure due to AssertionError
Closes #12425
2015-07-23 11:29:43 -04:00
Lee Hinman 18079168e4 Add shadow indicator when using shadow replicas 2015-07-23 09:13:54 -06:00
Christoph Büscher 026fd3843d Merge pull request #12342 from cbuescher/feature/query-refactoring-spanor
Query refactoring: SpanOrQueryBuilder and Parser
2015-07-23 17:10:39 +02:00
Robert Muir 4b2bda4195 make java.version mandatory for jvm plugins
JarHell has a low level check, but its more of a best effort one,
only checking if X-Compile-Target-JDK is set in the manifest. This
is the case for all lucene- and elasticsearch- generated jars, but
lets just be explicit for plugins.
2015-07-23 10:58:52 -04:00
khiraiwa 38d973751c Interval time is different from the doc value.
I think the value is 30.
2015-07-23 23:58:42 +09:00
Martijn van Groningen cafc7078e2 Removing TransportSingleCustomOperationAction in favour of TransportSingleShardAction to clean up code.
The TransportSingleCustomOperationAction `prefer_local` option has been removed as it isn't worth the effort.
The TransportSingleShardAction will execute the operation on the receiving node if a concrete list doesn't provide a list of candite shards routings to perform the operation on.
2015-07-23 16:42:10 +02:00
Adrien Grand 164d8c1ce2 Fix fielddata handling for the `_parent` field.
Change #12371 broke fielddata on the `_parent` child for indices created before
2.0. This pull request adds back caching of the `_parent` fielddata for indices
created before 2.0 and cleans some related stuff. For instance
DocumentTypeListener doesn't need to take care of removed mappings anymore since
mappings can't be removed in 2.0.
2015-07-23 14:13:21 +02:00
Martijn van Groningen 948da82f90 parent/child field data is an exception to the rule, because it is based on two fields... 2015-07-23 14:00:03 +02:00
Martijn van Groningen 2d775c34c9 field data: If a field doesn't exist in field info then always return a non cached empty atomic reader. 2015-07-23 14:00:03 +02:00
Robert Muir b0d8d5ed52 Merge pull request #12408 from rmuir/adapt_pluginmanager
Adapt pluginmanager to the new world
2015-07-23 07:16:59 -04:00
Robert Muir ea399065ab improve exception messages 2015-07-23 07:16:24 -04:00
Christoph Büscher 2b675ccf03 Merge branch 'master' into feature/query-refactoring 2015-07-23 12:32:26 +02:00
Christoph Büscher 3699119272 Remove unused QueryParseContext argument in MappedFieldType#rangeQuery()
The rangeQuery() method in MappedFieldType and some overwriting subtypes takes
a nullable QueryParseContext argument which is unused and can be deleted.
This is also useful for the current query parsing refactoring, since
we want to avoid passing the context object around unnecessarily.
2015-07-23 12:08:02 +02:00
Boaz Leskes 2d19868dd5 Test: await fix on IndicesShardStoreRequestTests.testCorruptedShards 2015-07-23 12:06:06 +02:00
Britta Weber 382daf26e4 Merge pull request #12386 from brwe/shard-active-test
[Test] make cluster state blocking more reliable in IndicesStoreIntegrationTests.indexCleanup()
2015-07-23 11:26:47 +02:00
Britta Weber d78cd66b51 [Test] make cluster state blocking more reliable in IndicesStoreIntegrationTests.indexCleanup()
IndicesStoreIntegrationTests.indexCleanup() tests if the shard files on disk are actually removed
after relocation. In particular it tests the following:
Whenever a node deletes a shard because it was relocated somewhere else, it first
checks if enough other copies are started somewhere else. The node sends a
ShardActiveRequest to the nodes that should have a copy.
The nodes that receive this request check if the shard is in state STARTED in which case
they respond with true. If they have the shard in POST_RECOVERY they register a cluster state
observer that checks at each update if the shard has moved to STARTED and respond with true when
this happens.

To test that the cluster state observer mechanism actually works, the latter can be triggered by
blocking the cluster state processing when a recover starts and only unblocking it shortly
after the node receives the ShardActiveRequest.

This is more reliable than using random cluster state processing delays
because the random delays make it hard to reason about different timeouts
that can be reached.

closes #11989
2015-07-23 11:26:15 +02:00
Shay Banon 33d2ca13a9 Simplify Replica Allocator
Simplify the codebase of replica allocator and add more unit tests for it
2015-07-23 07:47:39 +02:00
Robert Muir ed1642b968 remove obselete zips 2015-07-22 23:52:34 -04:00
Robert Muir b289af290e fix test to not use zips 2015-07-22 23:38:36 -04:00
Robert Muir 6f9a93f6bd complete jar hell checks 2015-07-22 22:19:44 -04:00
Robert Muir ae624acdc3 Use plugin descriptor file to cleanup pluginmanager logic and improve validation 2015-07-22 21:52:36 -04:00
Lee Hinman a33cfe4b11 Revert "Consistently name Groovy scripts with the same content"
This reverts commit d902012835.

Reverting this temporarily to see if this fixes hitting an assert in the
JDK during Groovy's classloading
2015-07-22 11:15:57 -06:00
Robert Muir 4040f194f5 Refactor pluginservice
Closes #12367

Squashed commit of the following:

commit 9453c411798121aa5439c52e95301f60a022ba5f
Merge: 3511a9c 828d8c7
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Jul 22 08:22:41 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit 3511a9c616503c447de9f0df9b4e9db3e22abd58
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:50:15 2015 -0700

    Remove duplicated constant

commit 4a9b5b4621b0ef2e74c1e017d9c8cf624dd27713
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:01:57 2015 -0700

    Add check that plugin must specify at least site or jvm

commit 19aef2f0596153a549ef4b7f4483694de41e101b
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:52:58 2015 -0700

    Change plugin "plugin" property to "classname"

commit 07ae396f30ed592b7499a086adca72d3f327fe4c
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:36:05 2015 -0400

    remove test with no methods

commit 550e73bf3d0f94562f4dde95239409dc5a24ce25
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:31:58 2015 -0400

    fix loading to use classname

commit 04463aed12046da0da5cac2a24c3ace51a79f799
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:24:19 2015 -0400

    rename to classname

commit 9f3afadd1caf89448c2eb913757036da48758b2d
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:18:46 2015 -0700

    moved PluginInfo and refactored parsing from properties file

commit df63ccc1b8b7cc64d3e59d23f6c8e827825eba87
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:08:26 2015 -0400

    fix test

commit c7febd844be358707823186a8c7a2d21e37540c9
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:03:44 2015 -0400

    remove test

commit 017b3410cf9d2b7fca1b8653e6f1ebe2f2519257
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:58:31 2015 -0400

    fix test

commit c9922938df48041ad43bbb3ed6746f71bc846629
Merge: ad59af4 01ea89a
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:37:28 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit ad59af465e1f1ac58897e63e0c25fcce641148a7
Author: Areek Zillur <areek.zillur@elasticsearch.com>
Date:   Tue Jul 21 19:30:26 2015 -0400

    [TEST] Verify expected number of nodes in cluster before issuing shardStores request

commit f0f5a1e087255215b93656550fbc6bd89b8b3205
Author: Lee Hinman <lee@writequit.org>
Date:   Tue Jul 21 11:27:28 2015 -0600

    Ignore EngineClosedException during translog fysnc

    When performing an operation on a primary, the state is captured and the
    operation is performed on the primary shard. The original request is
    then modified to increment the version of the operation as preparation
    for it to be sent to the replicas.

    If the request first fails on the primary during the translog sync
    (because the Engine is already closed due to shadow primaries closing
    the engine on relocation), then the operation is retried on the new primary
    after being modified for the replica shards. It will then fail due to the
    version being incorrect (the document does not yet exist but the request
    expects a version of "1").

    Order of operations:

    - Request is executed against primary
    - Request is modified (version incremented) so it can be sent to replicas
    - Engine's translog is fsync'd if necessary (failing, and throwing an exception)
    - Modified request is retried against new primary

    This change ignores the exception where the engine is already closed
    when syncing the translog (similar to how we ignore exceptions when
    refreshing the shard if the ?refresh=true flag is used).

commit 4ac68bb1658688550ced0c4f479dee6d8b617777
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 22:37:29 2015 +0200

    Replica allocator unit tests
    First batch of unit tests to verify the behavior of replica allocator

commit 94609fc5943c8d85adc751b553847ab4cebe58a3
Author: Jason Tedor <jason@tedor.me>
Date:   Tue Jul 21 14:04:46 2015 -0400

    Correctly list blobs in Azure storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage

    This commit addresses an issue that was leading to snapshot corruption for snapshots stored as blobs in Azure Storage.

    The underlying issue is that in cases when multiple snapshots of an index were taken and persisted into Azure Storage, snapshots subsequent
    to the first would repeatedly overwrite the snapshot files. This issue does render useless all snapshots except the final snapshot.

    The root cause of this is due to String concatenation involving null. In particular, to list all of the blobs in a snapshot directory in
    Azure the code would use the method listBlobsByPrefix where the prefix is null. In the listBlobsByPrefix method, the path keyPath + prefix
    is constructed. However, per 5.1.11, 5.4 and 15.18.1 of the Java Language Specification, the reference null is first converted to the string
    "null" before performing the concatenation. This leads to no blobs being returned and therefore the snapshot mechanism would operate as if
    it were writing the first snapshot of the index. The fix is simply to check if prefix is null and handle the concatenation accordingly.

    Upon fixing this issue so that subsequent snapshots would no longer overwrite earlier snapshots, it was discovered that the snapshot metadata
    returned by the listBlobsByPrefix method was not sufficient for the snapshot layer to detect whether or not the Lucene segments had already
    been copied to the Azure storage layer in an earlier snapshot. This led the snapshot layer to unnecessarily duplicate these Lucene segments
    in Azure Storage.

    The root cause of this is due to known behavior in the CloudBlobContainer.getBlockBlobReference method in the Azure API. Namely, this method
    does not fetch blob attributes from Azure. As such, the lengths of all the blobs appeared to the snapshot layer to be of length zero and
    therefore they would compare as not equal to any new blobs that the snapshot layer is going to persist. To remediate this, the method
    CloudBlockBlob.downloadAttributes must be invoked. This will fetch the attributes from Azure Storage so that a proper comparison of the
    blobs can be performed.

    Closes elastic/elasticsearch-cloud-azure#51, closes elastic/elasticsearch-cloud-azure#99

commit cf1d481ce5dda0a45805e42f3b2e0e1e5d028b9e
Author: Lee Hinman <lee@writequit.org>
Date:   Mon Jul 20 08:41:55 2015 -0600

    Unit tests for `nodesAndVersions` on shared filesystems

    With the `recover_on_any_node` setting, these unit tests check that the
    correct node list and versions are returned.

commit 3c27cc32395c3624f7c794904d9ea4faf2eccbfb
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 14:15:59 2015 -0400

    don't fail junit4 integration tests if there are no tests.

    instead fail the failsafe plugin, which means the external cluster will still get shut down

commit 95d2756c5a8c21a157fa844273fc83dfa3c00aea
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 17:16:53 2015 +0200

    Testing: Fix help displaying tests under windows

    The help files are using a unix based file separator, where as
    the test relies on the help being based on the file system separator.

    This commit fixes the test to remove all `\r` characters before
    comparing strings.

    The test has also been moved into its own CliToolTestCase, as it does
    not need to be an integration test.

commit 944f06ea36bd836f007f8eaade8f571d6140aad9
Author: Clinton Gormley <clint@traveljury.com>
Date:   Tue Jul 21 18:04:52 2015 +0200

    Refactored check_license_and_sha.pl to accept a license dir and package path

    In preparation for the move to building the core zip, tar.gz, rpm, and deb as separate modules, refactored check_license_and_sha.pl to:

    * accept a license dir and path to the package to check on the command line
    * to be able to extract zip, tar.gz, deb, and rpm
    * all packages except rpm will work on Windows

commit 2585431e8dfa5c82a2cc5b304cd03eee9bed7a4c
Author: Chris Earle <pickypg@users.noreply.github.com>
Date:   Tue Jul 21 08:35:28 2015 -0700

    Updating breaking changes

    - field names cannot be mapped with `.` in them
    - fixed asciidoc issue where the list was not recognized as a list

commit de299b9d3f4615b12e2226a1e2eff5a38ecaf15f
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 13:27:52 2015 +0200

    Replace primaryPostAllocated flag and use UnassignedInfo
    There is no need to maintain additional state as to if a primary was allocated post api creation on the index routing table, we hold all this information already in the UnassignedInfo class.
    closes #12374

commit 43080bff40f60bedce5bdbc92df302f73aeb9cae
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:45:05 2015 +0200

    PluginManager: Fix bin/plugin calls in scripts/bats test

    The release and smoke test python scripts used to install
    plugins in the old fashion.

    Also the BATS testing suite installed/removed plugins in that
    way. Here the marvel tests have been removed, as marvel currently
    does not work with the master branch.

    In addition documentation has been updated as well, where it was
    still missing.

commit b81ccba48993bc13c7678e6d979fd96998499233
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 11:37:50 2015 +0200

    Discovery: make sure NodeJoinController.ElectionCallback is always called from the update cluster state thread

    This is important for correct handling of the joining thread. This causes assertions to trip in our test runs. See http://build-us-00.elastic.co/job/es_g1gc_master_metal/11653/ as an example

    Closes #12372

commit 331853790bf29e34fb248ebc4c1ba585b44f5cab
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 15:54:36 2015 +0200

    Remove left over no commit from TransportReplicationAction

    It asks to double check thread pool rejection. I did and don't see problems with it.

commit e5724931bbc1603e37faa977af4235507f4811f5
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:31:57 2015 +0200

    CliTool: Various PluginManager fixes

    The new plugin manager parser was not called correctly in the scripts.
    In addition the plugin manager now creates a plugins/ directory in case
    it does not exist.

    Also the integration tests called the plugin manager in the deprecated way.

commit 7a815a370f83ff12ffb12717ac2fe62571311279
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 13:54:18 2015 +0200

    CLITool: Port PluginManager to use CLITool

    In order to unify the handling and reuse the CLITool infrastructure
    the plugin manager should make use of this as well.

    This obsolets the -i and --install options but requires the user
    to use `install` as the first argument of the CLI.

    This is basically just a port of the existing functionality, which
    is also the reason why this is not a refactoring of the plugin manager,
    which will come in a separate commit.

commit 7f171eba7b71ac5682a355684b6da703ffbfccc7
Author: Martijn van Groningen <martijn.v.groningen@gmail.com>
Date:   Tue Jul 21 10:44:21 2015 +0200

    Remove custom execute local logic in TransportSingleShardAction and TransportInstanceSingleOperationAction and rely on transport service to execute locally. (forking thread etc.)

    Change TransportInstanceSingleOperationAction to have shardActionHandler to, so we can execute locally without endless spinning.

commit 0f38e3eca6b570f74b552e70b4673f47934442e1
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 17:36:12 2015 -0700

    More readMetadata tests and pickiness

commit 880b47281bd69bd37807e8252934321b089c9f8e
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 14:42:09 2015 -0700

    Started unit tests for plugin service

commit cd7c8ddd7b8c4f3457824b493bffb19c156c7899
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 07:21:07 2015 -0400

    fix tests

commit 673454f0b14f072f66ed70e32110fae4f7aad642
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 06:58:25 2015 -0400

    refactor pluginservice
2015-07-22 10:45:45 -04:00
Christoph Büscher 8e20536f0f Adding utility method to SpanTermQueryBuilderTest that outputs certain amount of queries with same fieldname 2015-07-22 16:26:15 +02:00
Christoph Büscher afe4fb4a27 Remove empty check in doXContent, parser or validate will take care of this 2015-07-22 16:24:14 +02:00
Christoph Büscher 0135b68cec Added comments, some simplification to test 2015-07-22 16:24:14 +02:00
Christoph Büscher 885b258871 Query refactoring: SpanNearQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
2015-07-22 16:24:14 +02:00
Christoph Büscher fc1b5a993e Query refactoring: SpanWithinQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
2015-07-22 16:22:02 +02:00
Christoph Büscher e11e75e3ec Fix edge case for SpanMultiTermQueryBuilderTest 2015-07-22 15:56:25 +02:00
Christoph Büscher 7f88cc596b Query refactoring: SpanNotQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new doToQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #12365
2015-07-22 14:50:21 +02:00
Adrien Grand f8b741d705 Merge pull request #12371 from jpountz/fix/mapper_fielddata_dependency
Remove the dependecy on IndexFielddataService from MapperService.
2015-07-22 14:37:13 +02:00
Britta Weber 828d8c7754 Merge pull request #12328 from alexclare/issue-12327
Fix malformed query generation
2015-07-22 11:55:41 +02:00
Christoph Büscher 22dcc77062 Merge branch 'master' into feature/query-refactoring 2015-07-22 10:38:56 +02:00
Christoph Büscher 966c02b1ac Query refactoring: SpanMultiTermQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #12182
2015-07-22 10:36:15 +02:00
Areek Zillur 01ea89acbf [TEST] Verify expected number of nodes in cluster before issuing shardStores request 2015-07-21 19:30:26 -04:00
Lee Hinman c286cd11cf Ignore EngineClosedException during translog fysnc
When performing an operation on a primary, the state is captured and the
operation is performed on the primary shard. The original request is
then modified to increment the version of the operation as preparation
for it to be sent to the replicas.

If the request first fails on the primary during the translog sync
(because the Engine is already closed due to shadow primaries closing
the engine on relocation), then the operation is retried on the new primary
after being modified for the replica shards. It will then fail due to the
version being incorrect (the document does not yet exist but the request
expects a version of "1").

Order of operations:

- Request is executed against primary
- Request is modified (version incremented) so it can be sent to replicas
- Engine's translog is fsync'd if necessary (failing, and throwing an exception)
- Modified request is retried against new primary

This change ignores the exception where the engine is already closed
when syncing the translog (similar to how we ignore exceptions when
refreshing the shard if the ?refresh=true flag is used).
2015-07-21 15:09:40 -06:00
Shay Banon be3e315248 Replica allocator unit tests
First batch of unit tests to verify the behavior of replica allocator
2015-07-21 22:37:40 +02:00
Lee Hinman b230927dbc Unit tests for `nodesAndVersions` on shared filesystems
With the `recover_on_any_node` setting, these unit tests check that the
correct node list and versions are returned.
2015-07-21 14:17:51 -06:00
Alexander Reelsen 988535eb0d Testing: Fix help displaying tests under windows
The help files are using a unix based file separator, where as
the test relies on the help being based on the file system separator.

This commit fixes the test to remove all `\r` characters before
comparing strings.

The test has also been moved into its own CliToolTestCase, as it does
not need to be an integration test.
2015-07-21 18:53:55 +02:00
Shay Banon cd7096dfc2 Replace primaryPostAllocated flag and use UnassignedInfo
There is no need to maintain additional state as to if a primary was allocated post api creation on the index routing table, we hold all this information already in the UnassignedInfo class.
closes #12374
2015-07-21 17:00:07 +02:00
Alexander Reelsen d3e454780f PluginManager: Fix bin/plugin calls in scripts/bats test
The release and smoke test python scripts used to install
plugins in the old fashion.

Also the BATS testing suite installed/removed plugins in that
way. Here the marvel tests have been removed, as marvel currently
does not work with the master branch.

In addition documentation has been updated as well, where it was
still missing.
2015-07-21 16:17:59 +02:00
Boaz Leskes 5def4193e9 Discovery: make sure NodeJoinController.ElectionCallback is always called from the update cluster state thread
This is important for correct handling of the joining thread. This causes assertions to trip in our test runs. See http://build-us-00.elastic.co/job/es_g1gc_master_metal/11653/ as an example

Closes #12372
2015-07-21 16:11:37 +02:00
Boaz Leskes 7adf9929ba Remove left over no commit from TransportReplicationAction
It asks to double check thread pool rejection. I did and don't see problems with it.
2015-07-21 15:54:36 +02:00
Adrien Grand 3fd7a36f91 Remove the dependecy on IndexFielddataService from MapperService.
This dependency was used in order for mapping updates that change the fielddata
format to take effect immediately. And the way it worked was by clearing the
cache of fielddata instances that were already loaded. However, we do not need
to cache the already loaded (logical) fielddata instances, they are cheap to
regenerate. Note that the fielddata _caches_ are still kept around so that we
don't keep on rebuilding costly (physical) fielddata values.
2015-07-21 15:39:58 +02:00
Alexander Reelsen 999d5ab801 CliTool: Various PluginManager fixes
The new plugin manager parser was not called correctly in the scripts.
In addition the plugin manager now creates a plugins/ directory in case
it does not exist.

Also the integration tests called the plugin manager in the deprecated way.
2015-07-21 15:31:57 +02:00
Alexander Reelsen 2f54b89a23 CLITool: Port PluginManager to use CLITool
In order to unify the handling and reuse the CLITool infrastructure
the plugin manager should make use of this as well.

This obsolets the -i and --install options but requires the user
to use `install` as the first argument of the CLI.

This is basically just a port of the existing functionality, which
is also the reason why this is not a refactoring of the plugin manager,
which will come in a separate commit.
2015-07-21 14:15:39 +02:00
Ruflin 3be98927a0 Fix cidr mask conversion issue for 0.0.0.0/0 and add tests 2015-07-21 13:53:16 +02:00
Martijn van Groningen 0cf9c3268c Remove custom execute local logic in TransportSingleShardAction and TransportInstanceSingleOperationAction and rely on transport service to execute locally. (forking thread etc.)
Change TransportInstanceSingleOperationAction to have shardActionHandler to, so we can execute locally without endless spinning.
2015-07-21 13:45:42 +02:00
Alexander Reelsen 95f56d71bd Logging: Corrected potential confusing log message 2015-07-21 11:30:26 +02:00
Christoph Büscher 37cdc1344a Merge branch 'master' into feature/query-refactoring 2015-07-21 10:58:58 +02:00
Ryan Ernst e52a316609 Test: fix tests referring to _index 2015-07-21 01:24:49 -07:00
Ryan Ernst ededbf0c8b Merge pull request #12352 from rjernst/fix/index-name-in-mapper-service
Remove index name from mapping parser
2015-07-21 00:31:38 -07:00
Ryan Ernst 8cd03cce5e Merge branch 'master' into fix/12329 2015-07-21 00:29:34 -07:00
Boaz Leskes bb1318512b Test: move IndicesShardStoreRequestTests to the right package 2015-07-21 09:01:50 +02:00
Boaz Leskes 29c99ff23d Logging: shard store shouldn't log the entire cluster state under trace, but just its version.
If we want the entire CS, we can put the cluster.service on trace
2015-07-21 08:58:50 +02:00
Ryan Ernst 1c99626b84 Mappings: Remove ability to configure _index
The `_index` field is now a completely virtual field thanks
to #12027. It is no longer necessary to index the actual value
of the index name.

closes #12329
2015-07-20 23:54:35 -07:00
Ryan Ernst abcf269539 Remove old DocumentMapper.parse method 2015-07-20 22:37:44 -07:00
Areek Zillur ae750bca69 [TEST] add missing ensureYellow to IndicesShardStoreRequestTests.testBasic 2015-07-20 23:52:07 -04:00
Ryan Ernst 2ad682e8b4 Mappings: Remove index name from mapping parser
The index name was passed along through many levels of mapping parsing,
just so that it could be used for _index. However, the index name
is really metadata that should exist alongside things like type and
id in SourceToParse.

This change moves index name to SourceToParse, and eliminates it from the
DocumentMapperParser.
2015-07-20 17:50:07 -07:00
Robert Muir 052cf1446f Remove Environment.homeFile()
Today we grant read+write+delete access to any files underneath the home.
But we have to remove this, if we want to have improved security of files
underneath elasticsearch.
2015-07-20 20:27:36 -04:00
Martijn van Groningen 4e2bdb67b7 test: enable get field mapping tests 2015-07-20 23:38:47 +02:00
Areek Zillur e964b0c2ab [TEST] use official API to read cluster state and enable debug logging 2015-07-20 17:12:30 -04:00
Boaz Leskes 66e71da7bd Test: testTimeoutCleanup use assertBusy while checking timer queue 2015-07-20 22:38:53 +02:00
Jason Tedor 34155ff102 Merge pull request #12346 from jasontedor/fix/12345
Use time with nanosecond resolution calculated at the executing node to measure the time that contexts are held open
2015-07-20 15:54:54 -04:00
Boaz Leskes 068fae218f Test: testShardActiveElseWhere shouldn't force cluster state publishing
The test is supposed to fool a node, making it think all it's shards are allocated else where. The test verifies the node doesn't delete its shards. The current test submitted this "fake" CS to the master, causing it to published which in turn caused confusion on other nodes and test failures.

 See http://build-us-00.elastic.co/job/es_core_master_oracle_6/990/testReport/junit/org.elasticsearch.indices.store/IndicesStoreIntegrationTests/testShardActiveElseWhere/
2015-07-20 21:16:16 +02:00
Zachary Tong 9951db9c64 [TEST] Better floating point comparisons 2015-07-20 14:46:24 -04:00
Jason Tedor 3af763eb88 Use time with nanosecond resolution calculated at the executing node to measure the time that contexts are held open
Closes #12345
2015-07-20 12:42:13 -04:00
Martijn van Groningen 4c7bb17521 test: increased logging 2015-07-20 17:35:39 +02:00
Robert Muir cd9d1c617f Merge pull request #12344 from rmuir/2.1.16
Update randomizedtesting to 2.1.16
2015-07-20 11:13:57 -04:00
Robert Muir a0a6299840 Update randomizedtesting to 2.1.16 2015-07-20 11:02:50 -04:00
Shay Banon 32445bbc3a Simplify handling of ignored unassigned shards
Fold ignored unassigned to a UnassignedShards and have simpler handling of them. Also remove the trapy way of adding an ignored unassigned shards today directly to the list, and have dedicated methods for it.

This change also removes the useless moving of unassigned shards to the end, since anyhow we first, sort those unassigned shards, and second, we now have persistent "store exceptions" that should not cause "dead letter" shard allocation.
2015-07-20 15:43:31 +02:00
Shay Banon ca3e0c6d49 Initial Refactor Gateway Allocator
Break it into more manageable code by separating allocation primaries and allocating replicas. Start adding basic unit tests for primary shard allocator.
2015-07-20 14:04:24 +02:00
Christoph Büscher 7540ad5d50 Merge branch 'master' into feature/query-refactoring 2015-07-20 12:52:32 +02:00