Commit Graph

10699 Commits

Author SHA1 Message Date
Michael McCandless 1aad275c55 expose current CMS throttle in merge stats; fix tests, docs; also log per-merge stop/throttle/rate 2015-01-11 05:52:43 -05:00
Michael McCandless 31e6acf3f2 first cut 2015-01-10 16:38:56 -05:00
Boaz Leskes 102e7adfad Recovery: be more resilient to partial network partitions
This commits adds a test that simulate disconnecting nodes and dropping requests during the various stages of recovery and solves all the issues that were raised by it. In short:

1) On going recoveries will be scheduled for retry upon network disconnect. The default retry period is 5s (cross node connections are checked every 10s by default).
2) Sometimes the disconnect happens after the target engine has started (but the shard is still in recovery). For simplicity, I opted to restart the recovery from scratch (where little to no files will be copied again, because there were just synced).
3) To protected against dropped requests, a Recovery Monitor was added that fails a recovery if no progress has been made in the last 30m (by default), which is equivalent to the long time outs we use in recovery requests.
4) When a shard fails on a node, we try to assign it to another node. If no such node is available, the shard will remain unassigned, causing the target node to clean any in memory state for it (files on disk remain). At the moment the shard will remain unassigned until another cluster state change happens, which will re-assigned it to the node in question but if no such change happens the shard will remain stuck at unassigned. The commits adds an extra delayed reroute in such cases to make sure the shard will be reassinged
5) Moved all recovery related settings to the RecoverySettings.

Closes #8720
2015-01-10 15:19:30 +01:00
Michael McCandless dd8000c865 use Lucene's 4.10.3 version constant 2015-01-10 06:25:39 -05:00
Simon Willnauer e416ed2426 Don't pass indexing buffer side to the translog 2015-01-09 23:45:27 +01:00
Ryan Ernst 4cda543637 Tests: Add logic to handle static index upgrade case where index is
already on latest version.

See #9207
2015-01-09 12:40:44 -08:00
Robert Muir d226a973f7 core: upgrade to lucene 5 r1650327.
refactor _version docvalues migration to be more efficient.

closes #9206
2015-01-09 12:12:31 -05:00
Colin Goodheart-Smithe 91e00c6c8e Aggregations: Numeric metric aggregations are now formattable
You can now specify `format` in the request definition for most numeric metric aggregations. The exceptions are Percentile_Ranks, Cardinality and Value_Count as the response type of these can be different from the field type so the formatter won't work.

Closes #6812
2015-01-09 16:10:58 +00:00
Christoph Büscher 04cb09f44c [TEST] Add missing docs and tests for '_cat/segments'
The '_cat/segments' api was missing docs and a rest test which are added here.

Closes #5856
2015-01-09 12:29:11 +01:00
Simon Willnauer d2277d70ff [ENGINE] Simplify Engine construction and ref counting
Today the internal engine closes itself it the engine hits an exception
it can not recover from. This complicates a lot of refcounting issues
if such an exception happens during engine creation. This commit
only markes the engine as failed and let the user close it once the exception
bubbles up. Additionally it rolls back the indexwriter to prevent any changes after
the engine is failed.
2015-01-09 02:04:16 +01:00
Martijn van Groningen 592f517583 Serialize the rest status code, not the rest status enum. 2015-01-08 23:58:46 +01:00
David Pilato 6d58db8868 Mapping With a `null` Default Timestamp Causes NullPointerException on Merge
I have a field with a `null` [default `_timestamp` value](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-timestamp-field.html#mapping-timestamp-field-default) and when I try to update the mapping I get a server error caused by a `NullPointerException`

```
[2015-01-08 17:28:56,040][DEBUG][action.admin.indices.mapping.put] [...] failed to put mappings on indices [[feed_170_v1, feed_204_v1, feed_229_v1, feed_232_v1, feed_239_v1, feed_248_v1, feed_268_v1, feed_256_v1, feed_272_v1, feed_159_v1, feed_255_v1, feed_164_v1, feed_259_v1, feed_266_v1, feed_188_v1, feed_240_v1, feed_233_v1, feed_13_v1, feed_184_v1, feed_261_v1, feed_267_v1, feed_271_v1, feed_257_v1, feed_172_v1, feed_238_v1, feed_254_v1, feed_223_v1, feed_274_v1, feed_203_v1, feed_269_v1, feed_262_v1, feed_205_v1, feed_168_v1, feed_219_v1, feed_253_v1, feed_251_v1, feed_173_v1, feed_252_v1, feed_210_v1, feed_216_v1, feed_218_v1, feed_118_v1, feed_273_v1, feed_227_v1, feed_166_v1, feed_213_v1, feed_226_v1]], type [history]
java.lang.NullPointerException
        at org.elasticsearch.index.mapper.internal.TimestampFieldMapper.merge(TimestampFieldMapper.java:287)
        at org.elasticsearch.index.mapper.object.ObjectMapper.merge(ObjectMapper.java:936)
        at org.elasticsearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:693)
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$4.execute(MetaDataMappingService.java:508)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:329)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
```

https://github.com/elasticsearch/elasticsearch/blob/v1.4.2/src/main/java/org/elasticsearch/index/mapper/internal/TimestampFieldMapper.java#L286

Looks like the existence of default timestamp is not checked before use. The next line also has the same issue -- uses of default timestamp without checked to see if it's not null.

To reproduce:

```
$ curl -XPUT localhost:9200/twitter2

$ curl -XPUT localhost:9200/twitter2/tweet/_mapping -d '{
     "tweet" : {
         "_timestamp" : {
             "enabled" : true,
             "default" : null
         }
     }
}'

$ curl -XPUT localhost:9200/twitter2/tweet/_mapping -d '{
     "tweet" : {
         "_timestamp" : {
             "enabled" : true,
             "default" : null
         },
         "properties": {
             "user": {"type": "string"}
         }
     }
}'
```

Closes #9204.

(cherry picked from commit 62c6d63)
2015-01-08 21:33:17 +01:00
Ryan Ernst 7f9ffea97c Tests: Add upgrade step to static bwc tests 2015-01-08 11:53:48 -08:00
Ryan Ernst 060f963a8e Mappings: Remove allow_type_wrapper setting
Before Elasticsearch 1.0, the type was allowed to be passed as the root
element when uploading a document.  However, this was ambiguous if the
mappings also contained a field with the same name as the type.  The
behavior was changed in 1.0 to not allow this, but a setting was added
for backwards compatibility.  This change removes the setting for 2.0.
2015-01-08 09:13:40 -08:00
Martijn van Groningen ca4f27f40e Core: Added `_shards` header to all write responses.
The header indicates to how many shard copies (primary and replicas shards) a write was supposed to go to, to how many
shard copies to write succeeded and potentially captures shard failures if writing into a replica shard fails.

For async writes it also includes the number of shards a write is still pending.

Closes #7994
2015-01-08 18:10:08 +01:00
Ryan Ernst 1ad64a97ec Mappings: Remove includeExisting flag from adding ObjectMapper and
FieldMapper listeners

This flag is unused by the 2 places that add these listeners.
2015-01-08 09:08:54 -08:00
Simon Willnauer 959e3ca9da [CORE] Fold engine into IndexShard
This commit removes most of the Engine abstractions and removes
Engine exposure via dependency injection. It also removes the Holder
abstraction and makes the engine itself start at constrcution time.
It removes the start method from the engine entire which means no engine
instances exists until it's started. There is also no way to stop the
engine to restart, it needs to be an entire new Engine
2015-01-08 17:48:27 +01:00
Martijn van Groningen dedaf9387e Core: Also check if indices resolved via aliases resolution aren't closed and deal with this according to IndicesOptions.
Closes #9057
2015-01-08 16:45:34 +01:00
Martijn van Groningen b0b61ee0c3 Renamed allowNoIndices to failNoIndices and changed parameter order. 2015-01-08 16:43:56 +01:00
Simon Willnauer 78fc7c3f01 [TEST] Ensure shard lock is acquired before we try the timeout version 2015-01-08 15:37:31 +01:00
Colin Goodheart-Smithe ecfe72ebcc Indices API: Fix to make GET Index API consistent with docs
This fix ensures that calls to the GET alias/mappings/settings/warmers APIs return the aliases/mappings/settings/warmers object even if there is no content within them.. This make them consistent with the GET Index API docs and the breaking changes in 1.4 docs

Closes #9148
2015-01-08 08:48:44 +00:00
Boaz Leskes ad66d25fa2 Test: trace logging for testDeleteSafe 2015-01-07 23:16:19 +01:00
Simon Willnauer acf6132a99 Fix 1.5.0 Lucene version constant 2015-01-07 22:13:12 +01:00
Simon Willnauer 77493762e2 [TEST] Add back MockIndexEngine
This test class was lost accidentially in a8fa650
2015-01-07 16:38:21 +01:00
Adrien Grand fda727e20c Internal: assert that we do not call blocking code from transport threads.
This currently only adds checks to BaseFuture, but this should already cover
lots of client code. We could add more in the future, like interactions with
the filesystem and so on.

Close #9164
2015-01-07 14:08:40 +01:00
Martijn van Groningen 20f7be378b Removed parent parameter from update request, because it is just sets the routing.
The routing option should be used instead. The parent a child document points to can't be updated.

Closes #4538
2015-01-07 10:26:20 +01:00
Martijn van Groningen 687be70736 Made sure that named filters and queries defined in a wrapped query and filter are not lost.
Closes #6871
2015-01-07 09:10:16 +01:00
Martijn van Groningen c94d056454 Fixed a bug that was caused by specifying routing on a multi percolate request causing an ArrayIndexOutOfBoundsException.
The multi percolate shard responses are collected in an atomic array which uses the shard id is used as index, but the number of shards the multi percolate request was meant to go to was used as size of this array instead the total number of shards an index has. This caused the exception when routing was used.

Closes #6214
2015-01-07 08:49:25 +01:00
Simon Willnauer 7ec8973fbc [CORE] Delete shard content under lock
Once we delete the the index on a node we are closing all resources
and subsequently need to delete all shards contents from disk. Yet
this happens today under a lock (the shard lock) that needs to be
acquried in order to execute any operation on the shards data
path. We try to delete all the index meta-data once we acquired
all the shard lock but this operation can run into a timeout which causes
the index to remain on disk. Further, all shard data will be left on
disk if the timeout is reached.

This commit removes all the shards data just before the shard lock
is release as the last operation on a shard that belongs to a deleted
index.
2015-01-06 21:53:28 +01:00
Ryan Ernst f7f99b8dbf Stats: Added verbose option to segments api, with full ram tree as first
additional element per segment.

This commit adds a verbose flag to the _segments api.  Currently the
only additional information returned when set to true is the full
ram tree from lucene for each segment.
2015-01-06 10:04:52 -08:00
Simon Willnauer 09b4d0e097 [EXEC] Remove reduced stack size
We used to reduces the stack size to 256kb. This commit removes
this optimization and relies on the JVMs default.

Closes #9135
2015-01-06 17:35:45 +01:00
Adrien Grand bc86796592 Core: Remove terms filter cache.
This is our only cache which is not 'exact' and might allow for stalled results.
Additionally, a similar cache that we have and needs to perform lookups in other
indices in order to run queries is the script index, and for this index we rely
on the filesystem cache, so we should probably do the same with terms filters
lookups.

Close #9056
2015-01-06 17:21:20 +01:00
Simon Willnauer 236e2491b4 [ALLOCATION] Remove primary balance factor
The `cluster.routing.allocation.balance.primary` setting has caused
a lot of confusion in the past while it has very little benefit form a
shard allocatioon point of view. Users tend to modify this value to
evently distribute primaries across the nodes which is dangerous since
a prmiary flag on it's own can trigger relocations. The primary flag for a shard
is should not have any impact on cluster performance unless the high level feature
suffereing from primary hotspots is buggy. Yet, this setting was intended to be a
tie-breaker which is not necessary anymore since the algorithm is deterministic.

This commit removes this setting entriely.
2015-01-06 16:43:39 +01:00
Robert Muir 8948b489d6 core: Populate metadata.writtenBy for pre 1.3 index files.
Today this not populated (null) in these cases. But it would be useful to have
    this available, even just for improved error messages.

    The trickiest part today is the handling of 1.2.x files written with
    lucene 4.8+ which have both ES checksums and lucene ones. This is now simplified:
    when the lucene checksum is there, we always use it.

Closes #9152
2015-01-06 09:54:39 -05:00
Simon Willnauer 4900f52619 [ALLOCATION] Weight deltas must be absolute deltas
In some situations the shard balanceing weight delta becomes negative. Yet,
a negative delta is always treated as `well balanced` which is wrong. I wasn't
able to reproduce the issue in any way other than useing the real world data
from issue #9023. This commit adds a fix for absolute deltas as well as a base
test class that allows to build tests or simulations from the cat API output.

Closes #9023
2015-01-06 15:48:44 +01:00
Martijn van Groningen ca68136628 Made the `nested`, `reverse_nested` and `children` aggs ignore unmapped nested fields or unmapped child / parent types.
Closes #8760
2015-01-06 12:14:15 +01:00
Adrien Grand 4cb23a0520 Search: Fix paging on strings sorted in ascending order.
For the comparator to work correctly, we need to give it the same value in
`setTopValue` as the value that it gave back in `value`.

Close #9136
2015-01-06 11:48:05 +01:00
Adrien Grand 999bec1243 Make DistributorDirectory not call fsync on sub directories and missing files.
Related to #9145
2015-01-06 11:15:42 +01:00
Boaz Leskes 9090e0381f Internal: AdapterActionFuture should not set currentThread().interrupt()
If someone blocks on it and it is interrupted, we throw an ElasticsearchIllegalStateException. We should not set Thread.currentThread().interrupt(); in this case because we already communicate the interrupt through an exception.

Similar to #9001

Closes #9141
2015-01-06 11:03:27 +01:00
Adrien Grand cd04851206 Recovery: RecoveryTarget does not fsync the right file name.
Close #9144
2015-01-06 09:24:18 +01:00
Adrien Grand 90f98579a2 Upgrade to Lucene 5.0.0-snapshot-1649544.
A couple of changes that triggerred a refactoring in Elasticsearch:

 - LUCENE-6148: Accountable.getChildResources returns a collection instead of
   a list.

 - LUCENE-6121: CachingTokenFilter now propagates reset(), as a result
   SimpleQueryParser.newPossiblyAnalyzedQuery has been fixed to not reset both
   the underlying stream and the wrapper (otherwise lucene would barf because of
   a doubl reset).

 - LUCENE-6119: The auto-throttle issue changed a couple of method
   names/parameters. It also made
   `UpdateSettingsTests.testUpdateMergeMaxThreadCount` dead slow so I muted this
   test until we clea up merge throttling to use LUCENE-6119.

Close #9145
2015-01-06 09:24:18 +01:00
Alexander Reelsen 8626c18ad9 Settings: Ensure fields are overriden and not merged when using arrays
In the case you try to merge two settings, one being an array and one being
a field, together, the settings were merged instead of being overridden.

First config
my.value: 1

Second config
my.value: [ 2, 3 ]

If you execute

settingsBuilder().put(settings1).put(settings2).build()

now only values 2,3 will be in the final settings

Closes #8381
2015-01-06 09:13:10 +01:00
tlrx a5127d2ffd Plugins: Installation failed when bin/ and plugins/ directories are on different filesystems
Plugin installation failed when bin/, conf/ and plugins/ directories are on different file systems. The method File.move() can't be used to move a non-empty directory between different filesystems.

I didn't find a simple way to unittest that, even with in-memory filesystems like jimfs or the Lucene test framework.

Closes #8999
2015-01-06 08:45:59 +01:00
Robert Muir 9f6a6a832f fix TODO for master, we don't need to support this version here 2015-01-05 15:31:49 -05:00
Robert Muir 8f2f2c5663 Tests: add 0.20 index and fix test bugs in assertNewReplicasWork() 2015-01-05 15:30:18 -05:00
Clinton Gormley 0d9fad79e0 Fixed typo in geoshape exception
Invaild -> Invalid
2015-01-05 13:27:54 +01:00
Clinton Gormley 75cc7077c7 Update plugins.asciidoc
Added entity resolution plugin for duplication detection

Related to #9131
2015-01-05 12:53:37 +01:00
Britta Weber 9454593d6a [TEST] mute ExceptionRetryTests 2015-01-03 18:40:19 +01:00
David Pilato a50d82c44b [Doc] Use byte[] as example instead of String
Closes #8973.
2015-01-02 16:20:32 +01:00
Britta Weber f45e6ae3f9 [index] Prevent duplication of documents when retry indexing after fail
If bulk index request fails due to a disconnect, unavailable shard etc, the request is
retried once before actually failing. However, even in case of failure the documents
might already be indexed. For autogenerated ids the request must not add the
documents again and therfore canHaveDuplicates must be set to true.

closes #8788
2015-01-02 15:44:47 +01:00