Commit Graph

727 Commits

Author SHA1 Message Date
Robert Muir e472cbed09 Merge branch 'master' into perms
Conflicts:
	core/src/test/java/org/elasticsearch/plugins/PluginManagerTests.java
2015-08-03 13:20:34 -04:00
Robert Muir 0e027d82c9 Merge pull request #12602 from rmuir/core_integ_tests
Add integration test phase for elasticsearch core/
2015-08-03 10:15:37 -04:00
Robert Muir 379ffdb8ee Add integration test phase for elasticsearch core/ 2015-08-03 09:43:11 -04:00
Britta Weber 26d51c299e Merge pull request #12400 from brwe/plug-fetch-sub-phases
Make fetch sub phases pluggable
2015-08-03 15:19:28 +02:00
Adrien Grand 1eaa9f9fda Fix concurrency issue in PrioritizedEsThreadPoolExecutor.
Tasks can be registered with a timeout, which runs as a task in a separate
threadpool. The idea is that the timeout runner cancels the main task when
the time is out, and the timeout runner is cancelled when the main task
starts executing. However, the following statement:

```java
                    timeoutFuture = timer.schedule(new Runnable() {
                        @Override
                        public void run() {
                            if (remove(TieBreakingPrioritizedRunnable.this)) {
                                runAndClean(timeoutCallback);
                            }
                        }
                    }, timeValue.nanos(), TimeUnit.NANOSECONDS);
```

is not atomic: the removal task is first started, and then the (volatile)
variable is assigned. As a consequence, there is a short window that allows
a timeout task to wait until the time is out even if the task is already
completed.

See http://build-us-00.elastic.co/job/es_core_17_centos/496/ for an example of
such a failure.
2015-08-03 14:11:49 +02:00
Robert Muir c5b91b9058 Merge pull request #12589 from rmuir/shading_cleanup
Fix shaded jar packaging
2015-08-03 08:10:50 -04:00
Adrien Grand 365a324fc5 Tests: Add more logging to SearchPreferenceTests so that we can know which preference is failing. 2015-08-03 12:55:15 +02:00
Colin Goodheart-Smithe ade3881152 Aggregations: Full path validation for pipeline aggregations
Previously only the first aggregation in a buckets_path was check to make sure the aggregation existed. Now the whole path is checked to ensure an aggregation exists at each element in the buckets_path

Closes #12360
2015-08-03 11:15:40 +01:00
Robert Muir a5a27a5cd9 Add jar hell check test for the shaded jar 2015-08-03 05:54:55 -04:00
Robert Muir 20ed7c1724 Merge pull request #12586 from rmuir/exit_value
Fix Bootstrap to not call System.exit
2015-08-03 05:27:08 -04:00
Martijn van Groningen 50c345238b fix ShardUtils#getElasticsearchDirectoryReader(...) to use FilterDirectoryReader#getDelegate() 2015-08-03 09:54:56 +02:00
szroland 742607ec9b using default timezone explicitly 2015-08-02 20:53:03 +02:00
szroland 35d947b524 Add option to `_cat/indices` to return index creation date #11524
Returning index creation date, both as a numeric millisecond value and
as a string. This implements #11524
2015-08-02 20:53:02 +02:00
Robert Muir 6f33ea4f62 move shading to distribution/shaded 2015-08-01 11:37:43 -04:00
Robert Muir d173b89d11 make this less syntactically confusing 2015-07-31 14:13:00 -04:00
Robert Muir 5f9edd26ea Fix Bootstrap to not call System.exit
Its not going to work: its blocked by security policy
and will just add a confusing SecurityException to the mix, and
bogusly give an exit status of 0 when in fact something bad happened.

Finally, if ES can't startup, it is a serious problem, there is
no sense in hiding the reason why: deliver the full stack trace.
2015-07-31 14:00:18 -04:00
Robert Muir 8d5b5ad862 restore my lost code changes and add measures to detect/prevent this in the future 2015-07-31 13:00:24 -04:00
Igor Motov 354504334f Create a directory during repository verification
The repository verification process should create a subdirectory to make sure we check permission of newly created directories in case elasticsearch processes on different nodes are running using different uids and creating blobs with incompatible permissions.

Closes #11611
2015-07-31 12:11:05 -04:00
Robert Muir f804ec3cd9 cleanup 2015-07-31 10:05:11 -04:00
Robert Muir fd8e92a18a don't represent site plugins with 'null' anymore 2015-07-31 09:41:39 -04:00
Lee Hinman 5a8356c86e Reroute shards when a node goes under disk watermarks
Previously we issued a reroute when a node went over the high watermark
in order to move shards away from the node. This change tracks nodes
that have previously been over the high or low watermarks and issues a
reroute when the node goes back underneath the watermark.

This allows shards that may be unassigned to be assigned back to a node
that was previously over the low watermark but no longer is.

Resolves #12422
2015-07-30 16:32:54 -06:00
Robert Muir ba63d57e82 don't lose stacktrace if a plugin somehow makes jar hell 2015-07-30 15:56:33 -04:00
Britta Weber 8a4957967b more review comments 2015-07-30 17:47:21 +02:00
Alexander Reelsen a8310d0575 Tests: Fix CLI help checking tests under windows
As windows has different line endings and this has
already been fixed by another test, the method has been
moved into CliToolTestCase.

In addition one test has been removed, as it was redundant.
2015-07-30 16:12:14 +02:00
Alexander Reelsen 69d7f1a78a Startup: Remove getopt parsing in shell script, use java CLITool
In order to ensure, we have the same experience across operating systems
and shells, this commit uses the java CLI parser instead of the shell
getopt parsing to parse arguments.

This also allows for support for paths, which contain spaces.

Also commons-cli depdency was upgraded to 1.3.1 and tests have been added.

Changes

* new exit code, OK_AND_EXIT, allowing to tell the caller to exit, as everything
  went as expected (e.g. when running a version output)

BWC breaking:

* execute() returns an ExitStatus instead of an integer, otherwise there is no
  possibility to signal by a command, if the JVM should be exited after a run.
  This affects plugins, that have command line tools
* -v used to be version, but is a verbose flag by default in the current CLI infra,
  must be -V or --version now
* -X has been removed - the current implementation was useless anyway, as
  it prefixed those properties with "es.". You should use
  ES_JAVA_OPTS/JAVA_OPTS for JVM configuration
2015-07-30 13:20:29 +02:00
Martijn van Groningen b3b34668e8 made `dateMathExpressionResolver` final 2015-07-30 11:34:10 +02:00
Adrien Grand 23fa535bfa Upgrade HDRHistogram to version 2.1.6.
This release fixes https://github.com/HdrHistogram/HdrHistogram/pull/68 which
we have been hitting in our CI tests, for instance:
http://build-us-00.elastic.co/job/es_core_master_metal/10567/
2015-07-30 09:55:12 +02:00
Dave Parfitt 2d2f21e760 Merge pull request #12534 from metadave/feature/cat_nodeattrs
Add _cat/nodeattrs API
2015-07-29 17:01:22 -04:00
Dave Parfitt f209809716 Add _cat/nodeattrs API
This provides a _cat/nodeattrs API call, which presents
custom node attributes in a denormalized table.

Closes #8000
2015-07-29 16:11:18 -04:00
Martijn van Groningen 0c53204b40 Added RealtimeRequest interface and let GetRequest, MultiGetRequest, TermVectorsRequest and MultiTermVectorsRequest implement it.
This is useful because now all requests that are realtime are now grouped under the same marker interface.
2015-07-29 21:47:11 +02:00
Robert Muir c6d6eaef4d Add a run.sh to run from current source code with debugger
The maven magic needed for this is now heavier
2015-07-29 14:21:43 -04:00
Britta Weber b86d07fc9a generic for sub classes also? 2015-07-29 17:37:11 +02:00
Martijn van Groningen ac3d090379 Added date math support in index names
Date math index name resolution enables you to search a range of time-series indices, rather than searching all of your time-series indices and filtering the the results or maintaining aliases. Limiting the number of indices that are searched reduces the load on the cluster and improves execution performance. For example, if you are searching for errors in your daily logs, you can use a date math name template to restrict the search to the past two days.

The added `ExpressionResolver` implementation that is responsible for resolving date math expressions in index names. This resolver is evaluated before wildcard expressions are evaluated.

The supported format: `<static_name{date_math_expr{date_format|timezone_id}}>` and the date math expressions must be enclosed within angle brackets. The `date_format` is optional and defaults to `YYYY.MM.dd`. The `timezone_id` id is optional too and defaults to `utc`.

The `{` character can be escaped by places `\\` before it.

Closes #12059
2015-07-29 17:33:55 +02:00
Britta Weber 6754a26d08 more generic? 2015-07-29 16:46:13 +02:00
Britta Weber d113b59e86 add SearchContext to innerParse method 2015-07-29 16:06:40 +02:00
Britta Weber 6295738d4c make Context generic so we don't have to cast 2015-07-29 16:00:37 +02:00
Britta Weber 24c4b27978 add some documentation 2015-07-29 15:44:58 +02:00
Britta Weber 3ee3238d54 remove unused imports and vars 2015-07-29 15:44:58 +02:00
Britta Weber ea14f675b2 remove hasFetchSubPhaseContext 2015-07-29 15:44:58 +02:00
Britta Weber df339f4cb9 toArray 2015-07-29 15:44:58 +02:00
Britta Weber 5f0f66908e make fielddata fields a plugin 2015-07-29 15:44:58 +02:00
Britta Weber bcebb61f25 make fetch sub phases pluggable 2015-07-29 15:44:58 +02:00
Alexander Reelsen 86ab892986 Releases: Added version numbers & bwc indices for 1.6.2/1.7.1 2015-07-29 12:31:39 +02:00
Martijn van Groningen 5d7ed70fa0 Merge pull request #12518 from martijnvg/top_hits/bug/protected_against_crazy_size
Protected against  `size` and `offset` larger than total number of document in a shard
2015-07-29 10:33:27 +02:00
Dave Parfitt dc597f0bf5 Merge pull request #12509 from metadave/dp_snake_case_dates
Default date formats to use underscores via PUT
2015-07-28 16:57:45 -04:00
Lee Hinman 2dc4550839 Add more debugging information to the Awareness Decider
Relates to #12431
2015-07-28 13:48:40 -06:00
Jason Tedor 37177d2f4c Use consistent check whether or not timeout is set 2015-07-28 15:44:55 -04:00
Martijn van Groningen 3f6e470810 top_hits: If topN (based on `offset` + `size`) is higher than the maxDoc of an shard then normalize topN to maxDoc.
Closes #12510
2015-07-28 21:38:58 +02:00
Nik Everett d8a6ddf580 Merge _all: Stop NPE querying _all when it doesn't exist 2015-07-28 10:18:17 -04:00
Dave Parfitt af17b9cd40 Use underscores for date formats
@clintongormley

Closes #12429
2015-07-28 10:04:28 -04:00
Nik Everett adcd1fc11d _all: Add missing boost
When we rewrite to a MatchNoTermsQuery we were throwing out the boost which
could could lead to funky changes when the query against _all was in a
bool query.
2015-07-28 09:25:06 -04:00
Colin Goodheart-Smithe 2ebf229632 Aggregations: Fixes serialization of HDRHistogram in percentiles aggregations
Previously we would write the entire ByteBuffer to the stream to serialise the HDRHistogram even if it was not all needed. Now we only write the bytes that are actually written to in the ByteBuffer.
2015-07-28 14:07:06 +01:00
Colin Goodheart-Smithe 7ab9bbbfad [Test] Muted all HDRPercentilesTests and HDRPercentileRanksTests 2015-07-28 10:17:47 +01:00
Nik Everett d7491515b2 _all: Stop NPE querying _all when it doesn't exist
This can happen in two ways:
1. The _all field is disabled.
2. There are documents in the index, the _all field is enabled, but there are
no fields in any of the documents.

In both of these cases we now rewrite the query to a MatchNoDocsQuery which
should be safe because there isn't anything to match.

Closes #12439
2015-07-27 16:23:32 -04:00
Boaz Leskes 616941e902 Internal: IndicesStore shouldn't try to delete index after deleting a shard
When a node discovers shard content on disk which isn't used, we reach out to all other nodes that supposed to have the shard active. Only once all of those have confirmed the shard active, the shard has no unassigned copies *and* no cluster state change have happened in the mean while, do we go and delete the shard folder.

Currently, after removing a shard, the IndicesStores checks the indices services if that has no more shard active for this index and if so, it tries to delete the entire index folder (unless on master node, where we keep the index metadata around). This is wrong as both the check and the protections in IndicesServices.deleteIndexStore make sure that there isn't any shard *in use* from that index. However, it may be the we erroneously delete other unused shard copies on disk, without the proper safety guards described above.

Normally, this is not a problem as the missing copy will be recovered from another shard copy on another node (although a shame). However, in extremely rare cases involving multiple node failures/restarts where all shard copies are not available (i.e., shard is red) there are race conditions which can cause all shard copies to be deleted.

Instead, we should change the decision to clean up an index folder to based on checking the index directory for being empty and containing no shards.
2015-07-27 16:09:25 -04:00
Lee Hinman b8b96679ea [TEST] Mark 2 HDRPercentileRanks tests with AwaitsFix 2015-07-27 11:40:20 -06:00
Jason Tedor cdd2da1e1c Check for existence of snapshot index file before attempting delete 2015-07-27 13:28:37 -04:00
Alexander Reelsen 9628d2632f Build: Split packages into submodules
This change creates a proper `distribution` modules in which we have today packaging for
all of our four current packages:

* zip
* tar.gz
* rpm
* deb

Licenes have moved into the distribution project as well. So have the config/ and the bin/ directory
from the core/ project.

The RPM package is now built, if rpmbuild exists.

The bats tests have been moved as well.

Also the zip distribution now executes the REST integration tests.
2015-07-27 17:50:54 +02:00
Tanguy Leroux 01ae730023 Remove Core Lib directory 2015-07-27 16:52:10 +02:00
Lee Hinman 5fdc6cfc24 Revert "Revert "Consistently name Groovy scripts with the same content""
This reverts commit a33cfe4b11.
2015-07-27 08:49:47 -06:00
Colin Goodheart-Smithe 26a299d0e7 [TEST] muted HDRPercentilesTests.testScript_ExplicitSingleValued_WithParams 2015-07-27 15:42:47 +01:00
Adrien Grand a832987ada Forbid Files.isHidden.
As Robert pointed out on #12465, it has the undesirable property of relying on
the operating system. So it would be better to use a simple rule such as
checking whether the file name starts with a dot.
2015-07-27 16:13:04 +02:00
Martijn van Groningen a14913f7b6 Left over from the `query_cache` to `request_cache` rename. 2015-07-27 13:28:15 +02:00
Britta Weber 20facccaf6 Merge pull request #11666 from brwe/meta-data-zen
[TEST] remove redundant tests and move to different suite
2015-07-27 12:40:11 +02:00
Britta Weber e44c5ff703 check if index is closed or was previously closed when gathering relevant indices to write meta state
When an index is opened it will not be assigned to a node but also not have closed state
anymore. Before we only checked if an index either is closed or assigned to the data node
and therefore the change from close->open was not written.
2015-07-27 12:35:25 +02:00
Britta Weber 2713e903ab [TEST] remove redundant tests and move to different suite
Some of the test for meta data are redundant. Also, since they
somewhat test service disruptions (start master with empty
data folder) we might move them to DiscoveryWithServiceDisruptionsTests.
Also, this commit adds a test for
https://github.com/elastic/elasticsearch/issues/11665
2015-07-27 12:35:13 +02:00
Adrien Grand ec79568e6f Merge pull request #12430 from ruflin/ip_range-mask-fix
Fix cidr mask conversion issue for 0.0.0.0/0 and add tests #12005
2015-07-27 12:12:20 +02:00
javanna 8e931d5767 Query DSL: don't cache type filter in DocumentMapper
If we cache the type filter and we e.g. set its boost which is now settable on all queries, the boost will change for all subsequent queries. We should rather create a new query every time.
2015-07-27 11:35:19 +02:00
Ruflin 91d1789258 Move index creation to test setup method 2015-07-27 10:39:04 +02:00
Adrien Grand 00ae4bb792 Merge pull request #12465 from xuzha/xu-skip-hidden
skip hidden files
2015-07-27 10:06:45 +02:00
Ruflin b8c2f05ff5 Revert change to set longTo to "MAX_IP -1" and improve test suite to check for range 2015-07-27 09:11:04 +02:00
Shay Banon a99ccb6112 No need to find replica copy when index is created
There is no need to try and go fetch replica copies for best allocation when the index is created
2015-07-27 07:01:48 +09:00
xuzha b0b9c121c8 skip hidden files 2015-07-26 01:16:29 -07:00
Jason Tedor b3272fe648 Preserve the root cause when encountering an exception expecting to have reached the end of the setting stream 2015-07-25 15:29:15 -04:00
Martijn van Groningen c73fff799d Cleaned up the data structures used in MetaData for alias and index lookups.
Major changes:
* Changed MetaData to holds alias and index lookup information into a single TreeMap instead of two separate maps.
* Moved the building of the alias / index lookup to the metadata builder.
2015-07-24 22:04:11 +02:00
Jason Tedor 267afe866f Merge pull request #12451 from jasontedor/fix/12382
Add explicit check that we have reached the end of the settings stream when parsing settings
2015-07-24 13:52:00 -04:00
Jason Tedor 8ac105dbb9 Add explicit check that we have reached the end of the settings stream when parsing settings
Settings are currently parsed by looping over the tokens until an END_OBJECT token is reached. However, this does not mean that the end of
the settings stream was reached. This can occur, for example, when parsing a YAML settings file with inconsistent indentation. Currently
in this case, some settings will be silently ignored. This commit forces a check that we have in fact reached the end of the settings
stream.

Closes #12382
2015-07-24 13:12:44 -04:00
Colin Goodheart-Smithe 3e0532a0c5 Aggregations: Add HDRHistogram as an option in percentiles and percentile_ranks aggregations
HDRHistogram has been added as an option in the percentiles and percentile_ranks aggregation. It has one option `number_significant_digits` which controls the accuracy and memory size for the algorithm

Closes #8324
2015-07-24 17:55:36 +01:00
Adrien Grand e62aaa928e Tests: Fix TermVectorsUnitTests to not rely on HashMap iteration order. 2015-07-24 17:59:36 +02:00
Robert Muir 980e564caa Merge branch 'master' into plugin_java_version 2015-07-24 11:12:52 -04:00
Alex Ksikes 76c03df035 Make sure filter is correctly parsed for multi-term vectors
This makes sure the `filter` parameter is correctly parsed in a multi-term
vector request when using `ids` and `parameters`.

Closes #12311
Closes #12312
2015-07-24 17:09:16 +02:00
Boaz Leskes 57cbce2824 Allocation: ThrottlingAllocationDecider should not counting relocating shards
The ThrottlingAllocationDecider is responsible to limit the number of incoming/local recoveries on a node. It therefore shouldn't count shards marked as relocating which represent the source of the recovery.

Closes #12409
2015-07-24 16:29:51 +02:00
Martijn van Groningen e997342da4 Added IndexSearcherWrapper extension point.
This extension point allows one IndexSearcherWrapper instance to intercept the searcher from the Engine before it is used for a opertion.
2015-07-24 15:52:18 +02:00
Christoph Büscher 52113e7527 Merge pull request #12417 from cbuescher/remove-rangequery-context
Remove unused QueryParseContext argument in MappedFieldType#rangeQuery()
2015-07-24 14:19:53 +02:00
Adrien Grand 13d790bd4c Merge pull request #12423 from khiraiwa/patch-1
Interval time is different from the doc value.
2015-07-24 14:19:32 +02:00
Martijn van Groningen c781c89e2c ShardUtils#getElasticsearchLeafReader() should use FilterLeafReader#getDelegate() instead of FilterLeafReader#unwrap
If there are multiple levels of filtered leaf readers then with the unwrap() method it immediately returns the most inner leaf reader and thus skipping of over any other filtered leaf reader that may be instance of ElasticsearchLeafReader. By using #getDelegate() method we can check each filter reader layer if it is instance of ElasticsearchLeafReader, so that we never skip over any wrapped filtered leaf reader and lose the shard id.
2015-07-24 13:49:07 +02:00
Boaz Leskes 316d1cb6c0 Core: Adapt IndicesClusterStateService to use allocation ids
Also, as set of utility methods was introduced on ShardRouting to do various types of matching with other shard routings, giving control about what exactly should be matched (same shard id, same allocation id, all but version and shard info etc.). This is useful here, but also prepares the grounds for the change needed in #12387 (making ShardRouting.equals be strict and perform exact equality).

Closes #12397
2015-07-24 13:09:47 +02:00
Shay Banon fc90c2affd Cancel replica recovery on another sync option copy found
When a replica is initializing from the primary, and we find a better node that has full sync id match, it is better to cancel the existing replica allocation and allocate it to the new node with sync id match (eventually)
2015-07-24 12:13:03 +02:00
Jay Modi f8d9de8dca Merge pull request #12419 from jaymode/fix_settings_classloader
Copy the classloader from the original settings when checking for prompts
2015-07-23 13:06:09 -04:00
jaymode 8472775477 copy the classloader from the original settings when checking for prompts
Today, when a user provides settings and specifies a classloader to be used, the classloader gets
dropped when we copy the settings to check for prompt entries. This change copies the classloader
when replacing the prompt placeholders and adds a test to ensure the InternalSettingsPreparer
always retains the classloader.

Closes #12340
2015-07-23 12:53:27 -04:00
Jason Tedor 878301c795 Dump stack traces of all threads on test failure due to AssertionError
Closes #12425
2015-07-23 11:29:43 -04:00
Lee Hinman 18079168e4 Add shadow indicator when using shadow replicas 2015-07-23 09:13:54 -06:00
Robert Muir 4b2bda4195 make java.version mandatory for jvm plugins
JarHell has a low level check, but its more of a best effort one,
only checking if X-Compile-Target-JDK is set in the manifest. This
is the case for all lucene- and elasticsearch- generated jars, but
lets just be explicit for plugins.
2015-07-23 10:58:52 -04:00
khiraiwa 38d973751c Interval time is different from the doc value.
I think the value is 30.
2015-07-23 23:58:42 +09:00
Martijn van Groningen cafc7078e2 Removing TransportSingleCustomOperationAction in favour of TransportSingleShardAction to clean up code.
The TransportSingleCustomOperationAction `prefer_local` option has been removed as it isn't worth the effort.
The TransportSingleShardAction will execute the operation on the receiving node if a concrete list doesn't provide a list of candite shards routings to perform the operation on.
2015-07-23 16:42:10 +02:00
Adrien Grand 164d8c1ce2 Fix fielddata handling for the `_parent` field.
Change #12371 broke fielddata on the `_parent` child for indices created before
2.0. This pull request adds back caching of the `_parent` fielddata for indices
created before 2.0 and cleans some related stuff. For instance
DocumentTypeListener doesn't need to take care of removed mappings anymore since
mappings can't be removed in 2.0.
2015-07-23 14:13:21 +02:00
Martijn van Groningen 948da82f90 parent/child field data is an exception to the rule, because it is based on two fields... 2015-07-23 14:00:03 +02:00
Martijn van Groningen 2d775c34c9 field data: If a field doesn't exist in field info then always return a non cached empty atomic reader. 2015-07-23 14:00:03 +02:00
Robert Muir b0d8d5ed52 Merge pull request #12408 from rmuir/adapt_pluginmanager
Adapt pluginmanager to the new world
2015-07-23 07:16:59 -04:00
Robert Muir ea399065ab improve exception messages 2015-07-23 07:16:24 -04:00
Christoph Büscher 3699119272 Remove unused QueryParseContext argument in MappedFieldType#rangeQuery()
The rangeQuery() method in MappedFieldType and some overwriting subtypes takes
a nullable QueryParseContext argument which is unused and can be deleted.
This is also useful for the current query parsing refactoring, since
we want to avoid passing the context object around unnecessarily.
2015-07-23 12:08:02 +02:00
Boaz Leskes 2d19868dd5 Test: await fix on IndicesShardStoreRequestTests.testCorruptedShards 2015-07-23 12:06:06 +02:00
Britta Weber 382daf26e4 Merge pull request #12386 from brwe/shard-active-test
[Test] make cluster state blocking more reliable in IndicesStoreIntegrationTests.indexCleanup()
2015-07-23 11:26:47 +02:00
Britta Weber d78cd66b51 [Test] make cluster state blocking more reliable in IndicesStoreIntegrationTests.indexCleanup()
IndicesStoreIntegrationTests.indexCleanup() tests if the shard files on disk are actually removed
after relocation. In particular it tests the following:
Whenever a node deletes a shard because it was relocated somewhere else, it first
checks if enough other copies are started somewhere else. The node sends a
ShardActiveRequest to the nodes that should have a copy.
The nodes that receive this request check if the shard is in state STARTED in which case
they respond with true. If they have the shard in POST_RECOVERY they register a cluster state
observer that checks at each update if the shard has moved to STARTED and respond with true when
this happens.

To test that the cluster state observer mechanism actually works, the latter can be triggered by
blocking the cluster state processing when a recover starts and only unblocking it shortly
after the node receives the ShardActiveRequest.

This is more reliable than using random cluster state processing delays
because the random delays make it hard to reason about different timeouts
that can be reached.

closes #11989
2015-07-23 11:26:15 +02:00
Shay Banon 33d2ca13a9 Simplify Replica Allocator
Simplify the codebase of replica allocator and add more unit tests for it
2015-07-23 07:47:39 +02:00
Robert Muir ed1642b968 remove obselete zips 2015-07-22 23:52:34 -04:00
Robert Muir b289af290e fix test to not use zips 2015-07-22 23:38:36 -04:00
Robert Muir 6f9a93f6bd complete jar hell checks 2015-07-22 22:19:44 -04:00
Robert Muir ae624acdc3 Use plugin descriptor file to cleanup pluginmanager logic and improve validation 2015-07-22 21:52:36 -04:00
Lee Hinman a33cfe4b11 Revert "Consistently name Groovy scripts with the same content"
This reverts commit d902012835.

Reverting this temporarily to see if this fixes hitting an assert in the
JDK during Groovy's classloading
2015-07-22 11:15:57 -06:00
Robert Muir 4040f194f5 Refactor pluginservice
Closes #12367

Squashed commit of the following:

commit 9453c411798121aa5439c52e95301f60a022ba5f
Merge: 3511a9c 828d8c7
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Jul 22 08:22:41 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit 3511a9c616503c447de9f0df9b4e9db3e22abd58
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:50:15 2015 -0700

    Remove duplicated constant

commit 4a9b5b4621b0ef2e74c1e017d9c8cf624dd27713
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:01:57 2015 -0700

    Add check that plugin must specify at least site or jvm

commit 19aef2f0596153a549ef4b7f4483694de41e101b
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:52:58 2015 -0700

    Change plugin "plugin" property to "classname"

commit 07ae396f30ed592b7499a086adca72d3f327fe4c
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:36:05 2015 -0400

    remove test with no methods

commit 550e73bf3d0f94562f4dde95239409dc5a24ce25
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:31:58 2015 -0400

    fix loading to use classname

commit 04463aed12046da0da5cac2a24c3ace51a79f799
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:24:19 2015 -0400

    rename to classname

commit 9f3afadd1caf89448c2eb913757036da48758b2d
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:18:46 2015 -0700

    moved PluginInfo and refactored parsing from properties file

commit df63ccc1b8b7cc64d3e59d23f6c8e827825eba87
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:08:26 2015 -0400

    fix test

commit c7febd844be358707823186a8c7a2d21e37540c9
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:03:44 2015 -0400

    remove test

commit 017b3410cf9d2b7fca1b8653e6f1ebe2f2519257
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:58:31 2015 -0400

    fix test

commit c9922938df48041ad43bbb3ed6746f71bc846629
Merge: ad59af4 01ea89a
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:37:28 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit ad59af465e1f1ac58897e63e0c25fcce641148a7
Author: Areek Zillur <areek.zillur@elasticsearch.com>
Date:   Tue Jul 21 19:30:26 2015 -0400

    [TEST] Verify expected number of nodes in cluster before issuing shardStores request

commit f0f5a1e087255215b93656550fbc6bd89b8b3205
Author: Lee Hinman <lee@writequit.org>
Date:   Tue Jul 21 11:27:28 2015 -0600

    Ignore EngineClosedException during translog fysnc

    When performing an operation on a primary, the state is captured and the
    operation is performed on the primary shard. The original request is
    then modified to increment the version of the operation as preparation
    for it to be sent to the replicas.

    If the request first fails on the primary during the translog sync
    (because the Engine is already closed due to shadow primaries closing
    the engine on relocation), then the operation is retried on the new primary
    after being modified for the replica shards. It will then fail due to the
    version being incorrect (the document does not yet exist but the request
    expects a version of "1").

    Order of operations:

    - Request is executed against primary
    - Request is modified (version incremented) so it can be sent to replicas
    - Engine's translog is fsync'd if necessary (failing, and throwing an exception)
    - Modified request is retried against new primary

    This change ignores the exception where the engine is already closed
    when syncing the translog (similar to how we ignore exceptions when
    refreshing the shard if the ?refresh=true flag is used).

commit 4ac68bb1658688550ced0c4f479dee6d8b617777
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 22:37:29 2015 +0200

    Replica allocator unit tests
    First batch of unit tests to verify the behavior of replica allocator

commit 94609fc5943c8d85adc751b553847ab4cebe58a3
Author: Jason Tedor <jason@tedor.me>
Date:   Tue Jul 21 14:04:46 2015 -0400

    Correctly list blobs in Azure storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage

    This commit addresses an issue that was leading to snapshot corruption for snapshots stored as blobs in Azure Storage.

    The underlying issue is that in cases when multiple snapshots of an index were taken and persisted into Azure Storage, snapshots subsequent
    to the first would repeatedly overwrite the snapshot files. This issue does render useless all snapshots except the final snapshot.

    The root cause of this is due to String concatenation involving null. In particular, to list all of the blobs in a snapshot directory in
    Azure the code would use the method listBlobsByPrefix where the prefix is null. In the listBlobsByPrefix method, the path keyPath + prefix
    is constructed. However, per 5.1.11, 5.4 and 15.18.1 of the Java Language Specification, the reference null is first converted to the string
    "null" before performing the concatenation. This leads to no blobs being returned and therefore the snapshot mechanism would operate as if
    it were writing the first snapshot of the index. The fix is simply to check if prefix is null and handle the concatenation accordingly.

    Upon fixing this issue so that subsequent snapshots would no longer overwrite earlier snapshots, it was discovered that the snapshot metadata
    returned by the listBlobsByPrefix method was not sufficient for the snapshot layer to detect whether or not the Lucene segments had already
    been copied to the Azure storage layer in an earlier snapshot. This led the snapshot layer to unnecessarily duplicate these Lucene segments
    in Azure Storage.

    The root cause of this is due to known behavior in the CloudBlobContainer.getBlockBlobReference method in the Azure API. Namely, this method
    does not fetch blob attributes from Azure. As such, the lengths of all the blobs appeared to the snapshot layer to be of length zero and
    therefore they would compare as not equal to any new blobs that the snapshot layer is going to persist. To remediate this, the method
    CloudBlockBlob.downloadAttributes must be invoked. This will fetch the attributes from Azure Storage so that a proper comparison of the
    blobs can be performed.

    Closes elastic/elasticsearch-cloud-azure#51, closes elastic/elasticsearch-cloud-azure#99

commit cf1d481ce5dda0a45805e42f3b2e0e1e5d028b9e
Author: Lee Hinman <lee@writequit.org>
Date:   Mon Jul 20 08:41:55 2015 -0600

    Unit tests for `nodesAndVersions` on shared filesystems

    With the `recover_on_any_node` setting, these unit tests check that the
    correct node list and versions are returned.

commit 3c27cc32395c3624f7c794904d9ea4faf2eccbfb
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 14:15:59 2015 -0400

    don't fail junit4 integration tests if there are no tests.

    instead fail the failsafe plugin, which means the external cluster will still get shut down

commit 95d2756c5a8c21a157fa844273fc83dfa3c00aea
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 17:16:53 2015 +0200

    Testing: Fix help displaying tests under windows

    The help files are using a unix based file separator, where as
    the test relies on the help being based on the file system separator.

    This commit fixes the test to remove all `\r` characters before
    comparing strings.

    The test has also been moved into its own CliToolTestCase, as it does
    not need to be an integration test.

commit 944f06ea36bd836f007f8eaade8f571d6140aad9
Author: Clinton Gormley <clint@traveljury.com>
Date:   Tue Jul 21 18:04:52 2015 +0200

    Refactored check_license_and_sha.pl to accept a license dir and package path

    In preparation for the move to building the core zip, tar.gz, rpm, and deb as separate modules, refactored check_license_and_sha.pl to:

    * accept a license dir and path to the package to check on the command line
    * to be able to extract zip, tar.gz, deb, and rpm
    * all packages except rpm will work on Windows

commit 2585431e8dfa5c82a2cc5b304cd03eee9bed7a4c
Author: Chris Earle <pickypg@users.noreply.github.com>
Date:   Tue Jul 21 08:35:28 2015 -0700

    Updating breaking changes

    - field names cannot be mapped with `.` in them
    - fixed asciidoc issue where the list was not recognized as a list

commit de299b9d3f4615b12e2226a1e2eff5a38ecaf15f
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 13:27:52 2015 +0200

    Replace primaryPostAllocated flag and use UnassignedInfo
    There is no need to maintain additional state as to if a primary was allocated post api creation on the index routing table, we hold all this information already in the UnassignedInfo class.
    closes #12374

commit 43080bff40f60bedce5bdbc92df302f73aeb9cae
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:45:05 2015 +0200

    PluginManager: Fix bin/plugin calls in scripts/bats test

    The release and smoke test python scripts used to install
    plugins in the old fashion.

    Also the BATS testing suite installed/removed plugins in that
    way. Here the marvel tests have been removed, as marvel currently
    does not work with the master branch.

    In addition documentation has been updated as well, where it was
    still missing.

commit b81ccba48993bc13c7678e6d979fd96998499233
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 11:37:50 2015 +0200

    Discovery: make sure NodeJoinController.ElectionCallback is always called from the update cluster state thread

    This is important for correct handling of the joining thread. This causes assertions to trip in our test runs. See http://build-us-00.elastic.co/job/es_g1gc_master_metal/11653/ as an example

    Closes #12372

commit 331853790bf29e34fb248ebc4c1ba585b44f5cab
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 15:54:36 2015 +0200

    Remove left over no commit from TransportReplicationAction

    It asks to double check thread pool rejection. I did and don't see problems with it.

commit e5724931bbc1603e37faa977af4235507f4811f5
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:31:57 2015 +0200

    CliTool: Various PluginManager fixes

    The new plugin manager parser was not called correctly in the scripts.
    In addition the plugin manager now creates a plugins/ directory in case
    it does not exist.

    Also the integration tests called the plugin manager in the deprecated way.

commit 7a815a370f83ff12ffb12717ac2fe62571311279
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 13:54:18 2015 +0200

    CLITool: Port PluginManager to use CLITool

    In order to unify the handling and reuse the CLITool infrastructure
    the plugin manager should make use of this as well.

    This obsolets the -i and --install options but requires the user
    to use `install` as the first argument of the CLI.

    This is basically just a port of the existing functionality, which
    is also the reason why this is not a refactoring of the plugin manager,
    which will come in a separate commit.

commit 7f171eba7b71ac5682a355684b6da703ffbfccc7
Author: Martijn van Groningen <martijn.v.groningen@gmail.com>
Date:   Tue Jul 21 10:44:21 2015 +0200

    Remove custom execute local logic in TransportSingleShardAction and TransportInstanceSingleOperationAction and rely on transport service to execute locally. (forking thread etc.)

    Change TransportInstanceSingleOperationAction to have shardActionHandler to, so we can execute locally without endless spinning.

commit 0f38e3eca6b570f74b552e70b4673f47934442e1
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 17:36:12 2015 -0700

    More readMetadata tests and pickiness

commit 880b47281bd69bd37807e8252934321b089c9f8e
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 14:42:09 2015 -0700

    Started unit tests for plugin service

commit cd7c8ddd7b8c4f3457824b493bffb19c156c7899
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 07:21:07 2015 -0400

    fix tests

commit 673454f0b14f072f66ed70e32110fae4f7aad642
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 06:58:25 2015 -0400

    refactor pluginservice
2015-07-22 10:45:45 -04:00
Adrien Grand f8b741d705 Merge pull request #12371 from jpountz/fix/mapper_fielddata_dependency
Remove the dependecy on IndexFielddataService from MapperService.
2015-07-22 14:37:13 +02:00
Britta Weber 828d8c7754 Merge pull request #12328 from alexclare/issue-12327
Fix malformed query generation
2015-07-22 11:55:41 +02:00
Areek Zillur 01ea89acbf [TEST] Verify expected number of nodes in cluster before issuing shardStores request 2015-07-21 19:30:26 -04:00
Lee Hinman c286cd11cf Ignore EngineClosedException during translog fysnc
When performing an operation on a primary, the state is captured and the
operation is performed on the primary shard. The original request is
then modified to increment the version of the operation as preparation
for it to be sent to the replicas.

If the request first fails on the primary during the translog sync
(because the Engine is already closed due to shadow primaries closing
the engine on relocation), then the operation is retried on the new primary
after being modified for the replica shards. It will then fail due to the
version being incorrect (the document does not yet exist but the request
expects a version of "1").

Order of operations:

- Request is executed against primary
- Request is modified (version incremented) so it can be sent to replicas
- Engine's translog is fsync'd if necessary (failing, and throwing an exception)
- Modified request is retried against new primary

This change ignores the exception where the engine is already closed
when syncing the translog (similar to how we ignore exceptions when
refreshing the shard if the ?refresh=true flag is used).
2015-07-21 15:09:40 -06:00
Shay Banon be3e315248 Replica allocator unit tests
First batch of unit tests to verify the behavior of replica allocator
2015-07-21 22:37:40 +02:00
Lee Hinman b230927dbc Unit tests for `nodesAndVersions` on shared filesystems
With the `recover_on_any_node` setting, these unit tests check that the
correct node list and versions are returned.
2015-07-21 14:17:51 -06:00
Alexander Reelsen 988535eb0d Testing: Fix help displaying tests under windows
The help files are using a unix based file separator, where as
the test relies on the help being based on the file system separator.

This commit fixes the test to remove all `\r` characters before
comparing strings.

The test has also been moved into its own CliToolTestCase, as it does
not need to be an integration test.
2015-07-21 18:53:55 +02:00
Shay Banon cd7096dfc2 Replace primaryPostAllocated flag and use UnassignedInfo
There is no need to maintain additional state as to if a primary was allocated post api creation on the index routing table, we hold all this information already in the UnassignedInfo class.
closes #12374
2015-07-21 17:00:07 +02:00
Alexander Reelsen d3e454780f PluginManager: Fix bin/plugin calls in scripts/bats test
The release and smoke test python scripts used to install
plugins in the old fashion.

Also the BATS testing suite installed/removed plugins in that
way. Here the marvel tests have been removed, as marvel currently
does not work with the master branch.

In addition documentation has been updated as well, where it was
still missing.
2015-07-21 16:17:59 +02:00
Boaz Leskes 5def4193e9 Discovery: make sure NodeJoinController.ElectionCallback is always called from the update cluster state thread
This is important for correct handling of the joining thread. This causes assertions to trip in our test runs. See http://build-us-00.elastic.co/job/es_g1gc_master_metal/11653/ as an example

Closes #12372
2015-07-21 16:11:37 +02:00
Boaz Leskes 7adf9929ba Remove left over no commit from TransportReplicationAction
It asks to double check thread pool rejection. I did and don't see problems with it.
2015-07-21 15:54:36 +02:00
Adrien Grand 3fd7a36f91 Remove the dependecy on IndexFielddataService from MapperService.
This dependency was used in order for mapping updates that change the fielddata
format to take effect immediately. And the way it worked was by clearing the
cache of fielddata instances that were already loaded. However, we do not need
to cache the already loaded (logical) fielddata instances, they are cheap to
regenerate. Note that the fielddata _caches_ are still kept around so that we
don't keep on rebuilding costly (physical) fielddata values.
2015-07-21 15:39:58 +02:00
Alexander Reelsen 999d5ab801 CliTool: Various PluginManager fixes
The new plugin manager parser was not called correctly in the scripts.
In addition the plugin manager now creates a plugins/ directory in case
it does not exist.

Also the integration tests called the plugin manager in the deprecated way.
2015-07-21 15:31:57 +02:00
Alexander Reelsen 2f54b89a23 CLITool: Port PluginManager to use CLITool
In order to unify the handling and reuse the CLITool infrastructure
the plugin manager should make use of this as well.

This obsolets the -i and --install options but requires the user
to use `install` as the first argument of the CLI.

This is basically just a port of the existing functionality, which
is also the reason why this is not a refactoring of the plugin manager,
which will come in a separate commit.
2015-07-21 14:15:39 +02:00
Ruflin 3be98927a0 Fix cidr mask conversion issue for 0.0.0.0/0 and add tests 2015-07-21 13:53:16 +02:00
Martijn van Groningen 0cf9c3268c Remove custom execute local logic in TransportSingleShardAction and TransportInstanceSingleOperationAction and rely on transport service to execute locally. (forking thread etc.)
Change TransportInstanceSingleOperationAction to have shardActionHandler to, so we can execute locally without endless spinning.
2015-07-21 13:45:42 +02:00
Alexander Reelsen 95f56d71bd Logging: Corrected potential confusing log message 2015-07-21 11:30:26 +02:00
Ryan Ernst e52a316609 Test: fix tests referring to _index 2015-07-21 01:24:49 -07:00
Ryan Ernst ededbf0c8b Merge pull request #12352 from rjernst/fix/index-name-in-mapper-service
Remove index name from mapping parser
2015-07-21 00:31:38 -07:00
Ryan Ernst 8cd03cce5e Merge branch 'master' into fix/12329 2015-07-21 00:29:34 -07:00
Boaz Leskes bb1318512b Test: move IndicesShardStoreRequestTests to the right package 2015-07-21 09:01:50 +02:00
Boaz Leskes 29c99ff23d Logging: shard store shouldn't log the entire cluster state under trace, but just its version.
If we want the entire CS, we can put the cluster.service on trace
2015-07-21 08:58:50 +02:00
Ryan Ernst 1c99626b84 Mappings: Remove ability to configure _index
The `_index` field is now a completely virtual field thanks
to #12027. It is no longer necessary to index the actual value
of the index name.

closes #12329
2015-07-20 23:54:35 -07:00
Ryan Ernst abcf269539 Remove old DocumentMapper.parse method 2015-07-20 22:37:44 -07:00
Areek Zillur ae750bca69 [TEST] add missing ensureYellow to IndicesShardStoreRequestTests.testBasic 2015-07-20 23:52:07 -04:00
Ryan Ernst 2ad682e8b4 Mappings: Remove index name from mapping parser
The index name was passed along through many levels of mapping parsing,
just so that it could be used for _index. However, the index name
is really metadata that should exist alongside things like type and
id in SourceToParse.

This change moves index name to SourceToParse, and eliminates it from the
DocumentMapperParser.
2015-07-20 17:50:07 -07:00
Robert Muir 052cf1446f Remove Environment.homeFile()
Today we grant read+write+delete access to any files underneath the home.
But we have to remove this, if we want to have improved security of files
underneath elasticsearch.
2015-07-20 20:27:36 -04:00
Martijn van Groningen 4e2bdb67b7 test: enable get field mapping tests 2015-07-20 23:38:47 +02:00
Areek Zillur e964b0c2ab [TEST] use official API to read cluster state and enable debug logging 2015-07-20 17:12:30 -04:00
Boaz Leskes 66e71da7bd Test: testTimeoutCleanup use assertBusy while checking timer queue 2015-07-20 22:38:53 +02:00
Jason Tedor 34155ff102 Merge pull request #12346 from jasontedor/fix/12345
Use time with nanosecond resolution calculated at the executing node to measure the time that contexts are held open
2015-07-20 15:54:54 -04:00
Boaz Leskes 068fae218f Test: testShardActiveElseWhere shouldn't force cluster state publishing
The test is supposed to fool a node, making it think all it's shards are allocated else where. The test verifies the node doesn't delete its shards. The current test submitted this "fake" CS to the master, causing it to published which in turn caused confusion on other nodes and test failures.

 See http://build-us-00.elastic.co/job/es_core_master_oracle_6/990/testReport/junit/org.elasticsearch.indices.store/IndicesStoreIntegrationTests/testShardActiveElseWhere/
2015-07-20 21:16:16 +02:00
Zachary Tong 9951db9c64 [TEST] Better floating point comparisons 2015-07-20 14:46:24 -04:00
Jason Tedor 3af763eb88 Use time with nanosecond resolution calculated at the executing node to measure the time that contexts are held open
Closes #12345
2015-07-20 12:42:13 -04:00
Martijn van Groningen 4c7bb17521 test: increased logging 2015-07-20 17:35:39 +02:00
Robert Muir cd9d1c617f Merge pull request #12344 from rmuir/2.1.16
Update randomizedtesting to 2.1.16
2015-07-20 11:13:57 -04:00
Robert Muir a0a6299840 Update randomizedtesting to 2.1.16 2015-07-20 11:02:50 -04:00
Shay Banon 32445bbc3a Simplify handling of ignored unassigned shards
Fold ignored unassigned to a UnassignedShards and have simpler handling of them. Also remove the trapy way of adding an ignored unassigned shards today directly to the list, and have dedicated methods for it.

This change also removes the useless moving of unassigned shards to the end, since anyhow we first, sort those unassigned shards, and second, we now have persistent "store exceptions" that should not cause "dead letter" shard allocation.
2015-07-20 15:43:31 +02:00
Shay Banon ca3e0c6d49 Initial Refactor Gateway Allocator
Break it into more manageable code by separating allocation primaries and allocating replicas. Start adding basic unit tests for primary shard allocator.
2015-07-20 14:04:24 +02:00