9411 Commits

Author SHA1 Message Date
Martijn van Groningen
1849d0966c [Discovery] Made 'discovery.zen.rejoin_on_master_gone' setting updatable at runtime. 2014-08-27 15:46:46 +02:00
Martijn van Groningen
424a2f68c6 [Discovery] Removed METADATA block 2014-08-27 15:46:39 +02:00
Martijn van Groningen
4828e78637 [TEST] Added test that exposes a shard consistency problem when isolated node(s) rejoin the cluster after network segmentation and when the elected master node ended up on the lesser side of the network segmentation. 2014-08-27 15:46:39 +02:00
Martijn van Groningen
e7d24ecdd0 [TEST] Make sure there no initializing shards when network partition is simulated 2014-08-27 15:46:39 +02:00
Martijn van Groningen
fc8ae4d30d [TEST] Added test that verifies data integrity during and after a simulated network split. 2014-08-27 15:46:39 +02:00
Martijn van Groningen
2c9ef63676 [TEST] It may take a little bit before the unlucky node deals with the fact the master left 2014-08-27 15:46:38 +02:00
Boaz Leskes
d44bed5f48 [Internal] Do not execute cluster state changes if current node is no longer master
When a node steps down from being a master (because, for example, min_master_node is breached), it may still have
cluster state update tasks queued up. Most (but not all) are tasks that should no longer be executed as the node
no longer has authority to do so. Other cluster states updates, like electing the current node as master, should be
executed even if the current node is no longer master.

This commit make sure that, by default, `ClusterStateUpdateTask` is not executed if the node is no longer master. Tasks
that should run on non masters are changed to implement a new interface called `ClusterStateNonMasterUpdateTask`

Closes 
2014-08-27 15:46:38 +02:00
Boaz Leskes
a9aa10ade0 Updated to use ClusterBlocks new constructor signature
Introduced with: 11a3201a092ed6c5d31516ae4b30dbb618ba348c
2014-08-27 15:46:27 +02:00
Martijn van Groningen
2220c66535 [Discovery] Eagerly clean the routing table of shards that exist on nodes that are not in the latestDiscoNodes list.
Only the previous master node has been removed, so only shards allocated to that node will get failed.
This would have happened anyhow on later on when AllocationService#reroute is invoked (for example when a cluster setting changes or another cluster event),
but by cleaning the routing table pro-actively, the stale routing table is fixed sooner and therefor the shards
that are not accessible anyhow (because the node these shards were on has left the cluster) will get re-assigned sooner.
2014-08-27 15:46:23 +02:00
Martijn van Groningen
89a50f6013 [Discovery] If available newly elected master node should take over previous known nodes. 2014-08-27 15:46:23 +02:00
Martijn van Groningen
549076eb4c [Discovery] Changed the default for the 'rejoin_on_master_gone' option from false to true in zen discovery.
Added AwaitFix for the FullRollingRestartTests.
2014-08-27 15:46:14 +02:00
Martijn van Groningen
3cdbb1a79d [Discovery] Enable discovery.zen.rejoin_on_master_gone setting in DiscoveryWithNetworkFailuresTests only. 2014-08-27 15:46:10 +02:00
Martijn van Groningen
97bdc8f5a2 [Discovery] Make noMasterBlock configurable and added simple test that shows reads do execute (partially) when m_m_n isn't met 2014-08-27 15:45:34 +02:00
Shay Banon
6ede83ab45 [Discovery] add rejoin on master gone flag, defaults to false
defaults to false since there is still work left to properly make it work
2014-08-27 15:45:25 +02:00
Shay Banon
4824f05369 [Internal] make no master lock an instance var so it can be configured 2014-08-27 15:45:10 +02:00
Shay Banon
63d0406b67 [Discovery] lightweight minimum master node recovery
don't perform full recovery when minimum master nodes are not met, keep the state around and use it once elected as master
2014-08-27 15:45:02 +02:00
Lee Hinman
eaf392163c Add translog checksums
Switches TranslogStreams to check a header in the file to determine the
translog format, delegating to the version-specific stream.

Version 1 of the translog format writes a header using Lucene's
CodecUtil at the beginning of the file and appends a checksum for each
translog operation written.

Also refactors much of the translog operations, such as merging
.hasNext() and .next() in FsChannelSnapshot

Relates to 
2014-08-27 15:18:17 +02:00
Adrien Grand
b745b0151c Fielddata: Remove soft/resident caches.
These caches have no advantage compared to the default node cache. Additionally,
the soft cache makes use of soft references which make fielddata loading quite
unpredictable in addition to pushing more pressure on the garbage collector.

The `none` cache is still there because of tests. There is no other good
reason to use it.

LongFieldDataBenchmark has been removed because the refactoring exposed a
compilation error in this class, which seems to not having been working for a
long time. In addition it's not as much useful now that we are progressively
moving more fields to doc values.

Close 
2014-08-27 14:28:41 +02:00
Britta Weber
238efe505b bool query: parser should return match_all in case there are no clauses
This also fixes has_parent filters with a nested empty bool filter
(see test SimpleChildQuerySearchTests#test6722, the test should actually expect
either 0 results when searching for has_parent "test" or one result when
search for has_parent "foo")

closes 
closes 
2014-08-27 14:07:21 +02:00
Britta Weber
a92300c5b5 explain score: fix explanation streaming
Complex explanations were always read as Explanations. Depending
on if the response was streamed or not the explanation was
therefore generated by a ComplexExplanation or by a regular
Explanation.

closes 
2014-08-27 14:07:20 +02:00
javanna
92ae3c84fe Index templates: Made template filtering generic and extensible via plugins
Added the ability to register template filters that are being applied when a new index is created. The default filter that checks whether the template pattern matches the index name always runs first, additional filters can also be registered so that templates can be filtered out based on custom logic.

Took the chance to add the handy source(Object... source) method to PutIndexTemplateRequest and corresponding builder

Closes 
Closes 
2014-08-27 12:37:36 +02:00
Simon Willnauer
e4b7395026 [TEST] only bump replicas if we have enough nodes in the cluster 2014-08-27 12:14:45 +02:00
Colin Goodheart-Smithe
6797d73d7e [TEST] removed AwaitsFix, added checks to make sure indexed scripts are put correctly 2014-08-27 11:04:51 +01:00
Brian Murphy
6109ec36b5 Indexed Scripts : Change preference and thread option for GetRequest.
This change forces the GetRequest when a script is being loaded from an index
to use preference("_local") and threaded(false) to prevent the script service from
forking for GetRequests.
2014-08-27 10:45:53 +01:00
Simon Willnauer
5453c08f50 Use physical name to compare files from snapshot metadata
The comparison and read code in the BlobStoreIndexShardRepository
used the physicalName and Name in reverse order. This caused
SnapshotBackwardsCompatibilityTest to fail.

This reverts commit 636af40da1ae22d5b7c8a81435987768e9cc3ac9
2014-08-27 10:47:19 +02:00
Cristiano Fontes
ee46c3cd3f Mappings: Added support for empty field arrays
Close 
2014-08-27 10:17:05 +02:00
Martijn van Groningen
b6cdb1d8fb Parent/child: Add missing support for the field data loading option to the _parent field.
Closes 
Closes 
2014-08-27 09:04:42 +02:00
Martijn van Groningen
d414d89c62 Parent/child: If _parent field points to a non existing parent type, then skip the has_parent query/filter
Closes 
Closes 
2014-08-27 09:00:51 +02:00
Boaz Leskes
8a94044b69 [Test] testLargeClusterStatePublishing - bound max shard no to number of nodes and set replica count to 0
ensureGreen sometimes times out due to too many shards and GC kicking in
2014-08-27 08:34:19 +02:00
Ryan Ernst
1804f864d5 Internal: Add all unsafe variants of LZF compress library functions to forbidden APIs.
The "optimized" encoders/decoders have been unreliable and error prone.
Also, fix LZFCompressor.compress to use LZFEncoder.safeEncode, which
creates a new safe encoder, instead of using a shared encoder (which
is not threadsafe).

closes 
2014-08-26 20:17:07 -07:00
Ryan Ernst
c94c13fa26 Revert part of change in to fix issue because encoder is not threadsafe so cannot be shared 2014-08-26 14:04:59 -07:00
Ryan Ernst
d79c79c7d0 Internal: Add LZF safe encoder in LZFCompressor
Selecting the safe encoder fixes a 64bit JVM crash on big-endian architectures with
LZF UnsafeChunkEncoderBE.

closes 
2014-08-26 13:38:03 -07:00
Adrien Grand
636af40da1 Tests: Temporarily ignore SnapshotBackwardsCompatibilityTest 2014-08-26 18:13:36 +02:00
Adrien Grand
7623c5e401 Tests: Fix FileBasedMappingsTests by using the mappings API instead of field mappings. 2014-08-26 17:54:11 +02:00
Boaz Leskes
35b98f5c24 [Test] rewrite testNoMasterActions to use latest tooling
The test's timeout checks were thrown off by a client created randomly (when the timer was running).

Closes 
2014-08-26 17:48:24 +02:00
Britta Weber
b754d2b36b Test: mute test until we know what is going on 2014-08-26 15:42:24 +02:00
Simon Willnauer
c63626b537 [SNAPSHOT] Add BWC layer to .si / segments_N hashing
Due to additional safety added in  we compute now a strong hash for
.si and segments_N files which are compared during snapshot / restore.
Old snapshots don't have this hash which can cause unnecessary copying
of large amount of data. This commit adds the ability to fetch this
hash from the blob store if needed.

Closes 
2014-08-26 15:36:46 +02:00
Peter Johnson @insertcoffee
10197936da Docs: typo?
Closes 
2014-08-26 15:26:34 +02:00
Simon Willnauer
0676869e6d [ENGINE] Wait until engine is started up when acquireing searcher
Today we have a small window where a searcher can be acquired but the
engine is in the state of starting up. This causes a NPE triggering a
shard failure if we are fast enough. This commit fixes this situation
gracefully.

Closes 
2014-08-26 14:07:04 +02:00
Britta Weber
d7b8d1728e _all: report conflict on merge and throw exception on doc_values
- _all field was never merged when mapping was updated and no conflict reported
- _all accepted doc_values format although it is always tokenized

relates to 
closes 
2014-08-26 12:14:31 +02:00
mikemccand
075bd66713 Core: use Java's built-in ConcurrentHashMap
It's risky to have our own snapshot of Java 8's ConcurrentHashMap:
unless we keep the sources in sync over time (and OpenJDK's version
had already diverged), then we won't get bug/performance fixes.  Users
can choose to upgrade to Java 8 to see the improvements of CHM.

Closes 

Closes 
2014-08-26 06:11:05 -04:00
Alexander Reelsen
f2aa4a38bc Docs: Added link to clarify meaning of filtering in fielddata context 2014-08-26 12:00:06 +02:00
Adrien Grand
b43c2ced93 [TESTS] Temporary disable field data cache randomization. 2014-08-25 23:12:09 +02:00
mikemccand
783a9cbb18 Stats: add segments.index_writer_max_memory to see index writer's max RAM usage before buffered documents must be written to a new segment
Closes 

Closes 
2014-08-25 14:43:09 -04:00
Nik Everett
74287865b2 [Internal] discovery.id.seed is ignored
Closes , Closes 
2014-08-25 17:32:07 +02:00
javanna
3917ffc0ff [TEST] Explicitly clean up actions to be intercepted in IndicesRequestTests before asserting on collected requests
This helps making sure that no further requests are collected once we start asserting on them
2014-08-25 17:24:53 +02:00
Lee Hinman
1f7be7931b [TEST] fix issue clearing fielddata breaker introduced in 6950c38a0436ec937797f01fba8d7d95e6d6225f 2014-08-25 16:25:02 +02:00
Clinton Gormley
ce0b2ade9c Test: Fixed the regex in cat.indices/10_basic.yaml
And renamed the file from .yml to .yaml
2014-08-25 16:13:16 +02:00
Adrien Grand
2a67b129e2 [TESTS] Temporarily disable FileBasedMappingsTests. 2014-08-25 12:39:36 +02:00
markharwood
570c679420 Context suggester: infinite loop in GeolocationContextMapping
Close 
2014-08-25 11:56:39 +02:00