Commit Graph

24593 Commits

Author SHA1 Message Date
Boaz Leskes eaa105951f Simplify GlobalCheckpointService and properly hook it for cluster state updates (#20720)
During a recent merge from master, we lost the bridge from IndicesClusterStateService to the GlobalCheckpointService of primary shards, notifying them of changes to the current set of active/initializing shards. This commits add the bridge back (with unit tests). It also simplifies the GlobalCheckpoint tracking to use a simpler model (which makes use the fact that the global check point sync is done periodically).

The old integration CheckpointIT test is moved to IndexLevelReplicationTests. I also added similar assertions to RelocationsIT, which surfaced a bug in the primary relocation logic and how it plays with global checkpoint updates. The test is currently await-fixed and will be fixed in a follow up issue.
2016-10-17 16:33:03 +02:00
Ali Beyad 7c2e761c87 Sequence numbers commit data in Lucene uses Iterable interface (#20793)
Sequence number related data (maximum sequence number, local checkpoint,
and global checkpoint) gets stored in Lucene on each commit. The logical
place to store this data is on each Lucene commit's user commit data
structure (see IndexWriter#setCommitData and the new version
IndexWriter#setLiveCommitData). However, previously we did not store the
maximum sequence number in the commit data because the commit data got
copied over before the Lucene IndexWriter flushed the documents to segments
in the commit.  This means that between the time that the commit data was
set on the IndexWriter and the time that the IndexWriter completes the commit,
documents with higher sequence numbers could have entered the commit.
Hence, we would use FieldStats on the _seq_no field in the documents to get
the maximum sequence number value, but this suffers the drawback that if the
last sequence number in the commit corresponded to a delete document action,
that sequence number would not show up in FieldStats as there would be no
corresponding document in Lucene.

In Lucene 6.2, the commit data was changed to take an Iterable interface, so
that the commit data can be calculated and retrieved *after* all documents
have been flushed, while the commit data itself is being set on the Lucene commit.
This commit changes max_seq_no so it is stored in the commit data instead of
being calculated from FieldStats, taking advantage of the deferred calculation
of the max_seq_no through passing an Iterable that dynamically sets the iterator
data.

* improvements to iterating over commit data (and better safety guarantees)

* Adds sequence number and checkpoint testing for document deletion
intertwined with document indexing.

* improve test code slightly

* Remove caching of max_seq_no in commit data iterator and inline logging

* Adds a test for concurrently indexing and committing segments
to Lucene, ensuring the sequence number related commit data
in each Lucene commit point matches the invariants of
localCheckpoint <= highest sequence number in commit <= maxSeqNo

* fix comments

* addresses code review

* adds clarification on checking commit data on recovery from translog

* remove unneeded method
2016-10-12 12:38:26 -04:00
Jason Tedor 568033aba3 Fix typos in Javadoc in IndexShard.java
This commit fixes two typos on the Javadoc for
IndexShard#updateGlobalCheckpointOnPrimary.
2016-10-12 09:38:29 -04:00
Boaz Leskes 27eab74510 merge from master 2016-09-30 17:19:30 +02:00
Boaz Leskes 615928e8cd ESIndexLevelReplicationTestCase: Make it easier to add new TRA-based actions (#20708)
Right now our unit tests in that area only simulate indexing single documents. As we go forward it should be easy
to add other actions, like delete & bulk indexing. This commit extracts the common parts of the current indexing
logic to a based class make it easier to extend.
2016-09-30 15:49:39 +02:00
Tanguy Leroux bb73472107 Fix Setting.timeValue() methods (#20696)
The Setting.timeValue() method uses TimeValue.toString() which can produce fractional time values. These fractional time values cannot be parsed again by the settings framework.

This commit fix a method that still use the .toString() method and replaces it with .getStringRep(). It also changes a second method so that it's not up to the caller to decide which stringify method to call.

closes #20662
2016-09-30 15:30:44 +02:00
Martijn van Groningen 55dce523c2 docs: marked `foreach` processor as experimental
Closes #19602
2016-09-30 12:23:42 +02:00
Jason Tedor bfc6156a6d Fix failling logger level update test
This commit fixes a failing cluster settings tests, namely the logger
level update test. The test was incorrectly assuming the default log
level was info, but it could be non-info, for example, if
tests.es.logger.level is set to some non-info level.

Closes #20318
2016-09-30 08:36:13 +02:00
Jason Tedor 3a4ffd7b86 Fix failing logging listener tests
The logging listener tests started failing after
953a8a959b when the tests are run with
tests.es.logger.level set to any level other than debug. This is because
these tests were based around the assumption that the default logging
level was info, which was the case before that commit fixed setting the
default logging level via that system property. This commit fixes these
failing tests by adjusting this assumption to account for the fact that
the default logging level could be different.
2016-09-30 08:09:35 +02:00
Jason Tedor afcf683228 Remove ignore system bootstrap checks
Today we allow system bootstrap checks to be ignored with a
setting. Yet, the system bootstrap checks are as vital to the health of
a production node as the non-system checks (e.g., the original bootstrap
check, the file descriptor check, is critical for reducing the chances
of data loss from being too low). This commit removes the ability to
ignore system bootstrap checks.

Relates #20511
2016-09-30 02:18:54 +02:00
Simon Willnauer 7e3863d2d8 [TEST] Fix EvilSystemPropertyTests to be test order independent 2016-09-29 13:26:14 +02:00
Tal Levy 33b9e2065b no null values in ingest configuration error messages (#20616)
The invalid ingest configuration field name used to show itself,
even when it was null, in error messages. Sometimes this does not make
sense.

e.g.
```[null] Only one of [file], [id], or [inline] may be configure```
vs.
```Only one of [file], [id], or [inline] may be configure```

The above deals with three fields, therefore this no one property
responsible.
2016-09-29 11:34:52 +02:00
Simon Willnauer 74184cb1b0 Stabelize tests in phrase-suggest.asciidoc 2016-09-29 11:13:17 +02:00
Simon Willnauer f2e6862803 Add a hard limit for `index.number_of_shard` (#20682)
this change adds a hard limit to `index.number_of_shard` that prevents
indices from being created that have more than 1024 shards. This is still
a huge limit and can only be changed via settings a system property.
2016-09-29 11:03:30 +02:00
Jason Tedor 8879360f66 Fix failing doc tests in feature/seq_no
This commit fixes failing doc tests in feature/seq_no after merging
master into this branch.
2016-09-29 03:58:02 +02:00
Boaz Leskes a16d644c68 allow settings logging level via a sys config in unit tests
Pipe in the `tests.es.logger.level` system property to the log4j config file used in tests. We still default to info. Also adapts the logger name to use the first letter of packages.
2016-09-29 03:04:43 +02:00
Tal Levy 1872d2ae46 [fix] JSON Processor was not properly added (#20613) 2016-09-29 03:04:43 +02:00
Tal Levy f3a5ee671b [docs] [fix] `field` is no longer an option for the script processor (#20614) 2016-09-29 03:04:43 +02:00
Nicholas Knize 80c918a13a [TEST] Fix NumberFieldMapperTests.testNoDocValues to call correct helper method. 2016-09-29 03:04:43 +02:00
Jason Tedor 0808611184 Fix failing tests after merge
This commit fixes failing tests in feature/seq_no after merging master
in.
2016-09-29 03:04:37 +02:00
Boaz Leskes 953a8a959b allow settings logging level via a sys config in unit tests
Pipe in the `tests.es.logger.level` system property to the log4j config file used in tests. We still default to info. Also adapts the logger name to use the first letter of packages.
2016-09-29 01:33:13 +02:00
Jason Tedor 25fd9e26c4 Merge branch 'master' into feature/seq_no
* master: (1199 commits)
  [DOCS] Remove non-valid link to mapping migration document
  Revert "Default `include_in_all` for numeric-like types to false"
  test: add a test with ipv6 address
  docs: clearify that both ip4 and ip6 addresses are supported
  Include complex settings in settings requests
  Add production warning for pre-release builds
  Clean up confusing error message on unhandled endpoint
  [TEST] Increase logging level in testDelayShards()
  change health from string to enum (#20661)
  Provide error message when plugin id is missing
  Document that sliced scroll works for reindex
  Make reindex-from-remote ignore unknown fields
  Remove NoopGatewayAllocator in favor of a more realistic mock (#20637)
  Remove Marvel character reference from guide
  Fix documentation for setting Java I/O temp dir
  Update client benchmarks to log4j2
  Changes the API of GatewayAllocator#applyStartedShards and (#20642)
  Removes FailedRerouteAllocation and StartedRerouteAllocation
  IndexRoutingTable.initializeEmpty shouldn't override supplied primary RecoverySource (#20638)
  Smoke tester: Adjust to latest changes (#20611)
  ...
2016-09-29 00:22:31 +02:00
Tal Levy 92ab44d35c [fix] JSON Processor was not properly added (#20613) 2016-09-28 23:04:22 +02:00
Tal Levy 550a0449bc [docs] [fix] `field` is no longer an option for the script processor (#20614) 2016-09-28 21:50:32 +02:00
Nicholas Knize b1a508a3c8 [TEST] Fix NumberFieldMapperTests.testNoDocValues to call correct helper method. 2016-09-28 13:09:16 -05:00
Lee Hinman 85402d5220 [DOCS] Remove non-valid link to mapping migration document 2016-09-28 09:09:19 -06:00
Lee Hinman 3f77eacab1 Revert "Default `include_in_all` for numeric-like types to false"
This reverts commit 6666892038.
2016-09-28 07:07:46 -06:00
Martijn van Groningen c99890eda5 test: add a test with ipv6 address 2016-09-28 10:04:20 +02:00
Martijn van Groningen a7ad11c2e7 docs: clearify that both ip4 and ip6 addresses are supported 2016-09-28 10:03:43 +02:00
Jason Tedor 6013234560 Include complex settings in settings requests
Today when getting setting via an API like the cluster settings API,
complex settings are excluded (e.g.,
discovery.zen.ping.unicast.hosts). This commit adds these settings to
the output of such APIs.

Relates #20622
2016-09-27 20:25:03 -04:00
Jason Tedor 3c8ff45917 Add production warning for pre-release builds
This commit adds a usage warning when Elasticsearch is started with a
pre-release build.

Relates #20674
2016-09-27 20:13:12 -04:00
Lee Hinman 1e6a882ab9 Merge remote-tracking branch 'dakrone/cleanup-msg' 2016-09-27 13:12:50 -06:00
Lee Hinman e7ebfa0c69 Clean up confusing error message on unhandled endpoint
It currently returns something like:

```
"No feature for name [_siohgjoidfhjfihfg]"
```

Which is not the most understandable message, this changes it to be a
little more readable.

Resolves #10946
2016-09-27 08:42:47 -06:00
Lee Hinman e6e517a7a4 [TEST] Increase logging level in testDelayShards() 2016-09-27 03:01:38 -06:00
Martijn Laarman e5f1eb5dc9 change health from string to enum (#20661) 2016-09-26 22:44:30 +02:00
Lee Hinman 27496d6b92 Merge remote-tracking branch 'dakrone/breaker-done-resiliency' 2016-09-26 08:39:55 -06:00
Jason Tedor a6e33494ab Provide error message when plugin id is missing
Today when executing the install plugin command without a plugin id, we
end up throwing an NPE because the plugin id is null yet we just keep
going (ultimatley we try to lookup the null plugin id in a set, the
direct cause of the NPE). This commit modifies the install command so
that a missing plugin id is detected and help is provided to the user.

Relates #20660
2016-09-26 08:09:15 -04:00
Nik Everett 560fba1b28 Document that sliced scroll works for reindex
Surprise! You can use sliced scroll to easily parallelize reindex
and friend. They support it because they use the same infrastructure
as a regular search to parse the search request. While we would like
to make an "automatic" option for parallelizing reindex, this manual
option works right now and is pretty convenient!
2016-09-26 05:27:44 +02:00
Nik Everett 370afa371b Make reindex-from-remote ignore unknown fields
reindex-from-remote should ignore unknown fields so it is mostly
future compatible. This makes it ignore unknown fields by adding an
option to `ObjectParser` and `ConstructingObjectParser` that, if
enabled, causes them to ignore unknown fields.

Closes #20504
2016-09-26 00:55:46 +02:00
Boaz Leskes ee76c1a5c9 Remove NoopGatewayAllocator in favor of a more realistic mock (#20637)
Many of our unit tests instantiate an `AllocationService`, which requires having a `GatewayAllocator`. Today almost all of our test use a class called `NoopGatewayAllocator` which does nothing, effectively leaving all shard assignments to the balanced allocator. This is sad as it means we test a system that behaves differently than our production logic in very basic things. For example, a started primary that is lost will be assigned to a node that didn't use to have it.

This PR removes `NoopGatewayAllocator` in favor of a new `TestGatewayAllocator` that inherits the standard `GatewayAllocator` and overrides shard information fetching to return information based on historical assignments the allocator has done. The only exception is `BalanceConfigurationTests` which does test only the balancer and I opted to not have it work around the `GatewayAllocator` being in it's way.
2016-09-25 20:15:30 +02:00
Pius 660d9d0de3 Remove Marvel character reference from guide
Relates #20654
2016-09-25 11:28:13 -04:00
Jason Tedor a7c777f58a Fix documentation for setting Java I/O temp dir
This commit fixes the documentation for configuring the Java I/O temp
dir which incorrectly suggested using the -D flag as a parameter on the
command line; these flags have been removed and should now be specified
as arguments to the JVM using either the ES_JAVA_OPTS environment
variable or using the jvm.options configuration file.

Closes #20652
2016-09-24 21:13:22 -04:00
Daniel Mitterdorfer 7a1ad716fa Update client benchmarks to log4j2 2016-09-23 15:39:18 +02:00
Ali Beyad ac1b13dde7 Changes the API of GatewayAllocator#applyStartedShards and (#20642)
Changes the API of GatewayAllocator#applyStartedShards and 
GatewayAllocator#applyFailedShards to take both a RoutingAllocation
and a list of shards to apply. This allows better mock allocators
to be created as being done in #20637.

Closes #20642
2016-09-23 09:31:46 -04:00
Ali Beyad 029fc909b5 Removes FailedRerouteAllocation and StartedRerouteAllocation
Removes the FailedRerouteAllocation class and StartedRerouteAllocation
class, as they were just wrappers for RerouteAllocation that stored
started and failed shards, but these started and failed shards can
be passed in directly to the methods that needed them, removing the
need for this wrapper class and extra level of indirection.

Closes #20626
2016-09-23 09:02:36 -04:00
Boaz Leskes 65356697ac IndexRoutingTable.initializeEmpty shouldn't override supplied primary RecoverySource (#20638)
When initializing a new index routing table, we make a decision where the primary shards should be recovered from. This can be an empty folder for new indices, a set of specific allocation ids for old indices or a snapshot. We currently allow callers of `IndexRoutingTable.initializeEmpty` to supply the source but also set it automatically if null is given. Sadly the current logic is reusing the supplied parameter to store the result of the automatic decision. This is flawed if some of the decision should be *different* between the different index shard (as the first decision that is maid sticks).

 This commit fixes this but also simplifies the API to always make an automatic decision.

This was discovered while working on #20637 which strengthens the testing infra and caused this to bubble up. I put it as a separate commit to make sure it is not lost as part of a bigger test only PR.
2016-09-23 13:17:54 +02:00
Alexander Reelsen 56cc87b7cc Smoke tester: Adjust to latest changes (#20611)
* Removed the check for the correct hash in the `/` response, otherwise two hashes would be needed
* Adapted to new URLs
* Added two missing plugins
* Downloaded gpg key only once
2016-09-23 11:28:46 +02:00
Simon Willnauer fe1803c957 Remove AnalysisService and reduce it to a simple name to analyzer mapping (#20627)
Today we hold on to all possible tokenizers, tokenfilters etc. when we create
an index service on a node. This was mainly done to allow the `_analyze` API to
directly access all these primitive. We fixed this in #19827 and can now get rid of
the AnalysisService entirely and replace it with a simple map like class. This
ensures we don't create a gazillion long living objects that are entirely useless since
they are never used in most of the indices. Also those objects might consume a considerable
amount of memory since they might load stopwords or synonyms etc.

Closes #19828
2016-09-23 08:53:50 +02:00
Clinton Gormley e3b7b4f032 Reorganised docs for mapping safeguard settings 2016-09-22 14:58:17 +02:00
Tanguy Leroux ab2e067ef5 Make ByteSizeUnit implements Writeable (#20557)
This commit makes ByteSizeUnit implement Writeable.
2016-09-22 14:42:13 +02:00