Commit Graph

10717 Commits

Author SHA1 Message Date
Adrien Grand b8be8e432e Query cache: Make the query cache usable on time-based data.
The query cache has a mechanism that disables it automatically when
SearchContext.nowInMillis() is used. One issue with that is that the date math
parser always evaluates the current timestamp when parsing a date, even if it
is not needed. As a consequence, whenever you use a date expression in your
queries, the query cache would not be used.

Close #9225
2015-01-13 17:23:59 +01:00
Luca Cavanna 4bdc89fca0 Java API: package private getters to become public if there have corresponding public setters
Some of our Java API requests have public setters but their corresponding getters are package private only. This commit makes those getters public as well.

Closes #9273
2015-01-13 15:22:01 +01:00
Adrien Grand d583080f20 Remove unused DoubleObjectPagedHashMap. 2015-01-13 10:49:32 +01:00
David Pilato be1610ba63 [Mapper] Using default=null for _timestamp field creates a index loss on restart
Step to reproduce:

* Create new index and type.

```
DELETE new_index
PUT new_index
 {
    "mappings": {
    	"power": {
	        "_timestamp" : {
	            "enabled" : true,
	            "default": null
	        }
	    }
    }
}
```

* Add a document

```
PUT new_index/power/1
{
    "foo": "bar"
}
```

* Restart cluster ... and **index is missing**...

```
GET new_index
```

Gives IndexMissingException

Closes #9223.

(cherry picked from commit e654a2c)
(cherry picked from commit aef3bc2)
2015-01-13 09:49:26 +01:00
Ryan Ernst 48bc132927 Mappings: Remove fieldSearchAnalyzer and fieldSearchQuoteAnalyzer from
MapperService.

Instead, get the FieldMapper for the field and check the analyzer there.
2015-01-12 14:53:18 -08:00
Ryan Ernst c29c18f0f2 Internal: Log when upgrade starts and stops
closes #9227
2015-01-12 14:40:04 -08:00
Paul Echeverri 4f938ad37e Updates the command to add the repo to not use add-apt-repository, which automatically adds a non-working deb-src line to sources.list. Command now uses echo to write the correct line to sources.list instead.
Fixes #9261
2015-01-12 21:18:00 +00:00
Ryan Ernst 69f74d714f Tests: Fix failure introduced in #8967 2015-01-12 09:27:27 -08:00
Ryan Ernst 6f214d791f Tests: Fix test failure introduced in #8967 2015-01-12 08:42:59 -08:00
Ryan Ernst 7d5a15e461 Mapping: serialize doc values settings for _timestamp
This change fixes _timestamp's serialization method to write out
`doc_values` and `doc_values_format`, which could already be set,
but would not be written out.

closes #8893
closes #8967
2015-01-12 08:26:59 -08:00
Tomoya Hirano 15d46988dc Fix typo in sample json
Fixes #9253
2015-01-12 15:58:16 +00:00
Lee Hinman a3972f03c6 Pass index settings where appropriate in IndicesLifecycle
This allows plugins to be able to perform some needed setup before and
after an index/shard is in use.
2015-01-12 15:04:37 +01:00
Lee Hinman 6b24921bd4 [TEST] Re-add rebalance disable setting in RecoveryFromGatewayTests 2015-01-12 14:21:31 +01:00
David Pilato 052645903a Rest: remove status code from main action
Today we give the HTTP status back within the HTTP response itself and within the JSON response as well:

```sh
curl localhost:9200/
```

```js
{
  "status" : 200,
  "name" : "Red Wolf",
  "version" : {
    "number" : "2.0.0",
    "build_hash" : "6837a61d8a646a2ac7dc8da1ab3c4ab85d60882d",
    "build_timestamp" : "2014-08-19T13:55:56Z",
    "build_snapshot" : true,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}
```
2015-01-12 12:37:46 +01:00
tlrx 261eb5b5ce Windows: makes elasticsearch.bat more friendly to automated processes
On Windows platforms when JAVA_HOME is not defined, a message is printed on standard output and the bat script is paused until the user press a key. This behavior is not compliant to automated processes where elasticsearch.bat can be executed by another script. This commit adds a new parameter --silent / -s that allow to skip the pause. Also, the error message is directed to standard and error outputs.

 Closes #8913
2015-01-12 12:24:40 +01:00
David Pilato fc7a0d3a4a [Docs] fix three to four 2015-01-12 12:13:23 +01:00
Sebastian Utz 358bb9bd75 fixup! fixup! add support for registering custom circuit breaker 2015-01-12 10:32:14 +01:00
Martijn van Groningen a88dd36df4 Test: Only run the test that verifies the `pending` field in the `_shards` header if the test cluster has two are more data nodes. 2015-01-12 10:29:23 +01:00
Michael McCandless 1aad275c55 expose current CMS throttle in merge stats; fix tests, docs; also log per-merge stop/throttle/rate 2015-01-11 05:52:43 -05:00
Michael McCandless 31e6acf3f2 first cut 2015-01-10 16:38:56 -05:00
Boaz Leskes 102e7adfad Recovery: be more resilient to partial network partitions
This commits adds a test that simulate disconnecting nodes and dropping requests during the various stages of recovery and solves all the issues that were raised by it. In short:

1) On going recoveries will be scheduled for retry upon network disconnect. The default retry period is 5s (cross node connections are checked every 10s by default).
2) Sometimes the disconnect happens after the target engine has started (but the shard is still in recovery). For simplicity, I opted to restart the recovery from scratch (where little to no files will be copied again, because there were just synced).
3) To protected against dropped requests, a Recovery Monitor was added that fails a recovery if no progress has been made in the last 30m (by default), which is equivalent to the long time outs we use in recovery requests.
4) When a shard fails on a node, we try to assign it to another node. If no such node is available, the shard will remain unassigned, causing the target node to clean any in memory state for it (files on disk remain). At the moment the shard will remain unassigned until another cluster state change happens, which will re-assigned it to the node in question but if no such change happens the shard will remain stuck at unassigned. The commits adds an extra delayed reroute in such cases to make sure the shard will be reassinged
5) Moved all recovery related settings to the RecoverySettings.

Closes #8720
2015-01-10 15:19:30 +01:00
Michael McCandless dd8000c865 use Lucene's 4.10.3 version constant 2015-01-10 06:25:39 -05:00
Simon Willnauer e416ed2426 Don't pass indexing buffer side to the translog 2015-01-09 23:45:27 +01:00
Ryan Ernst 4cda543637 Tests: Add logic to handle static index upgrade case where index is
already on latest version.

See #9207
2015-01-09 12:40:44 -08:00
Robert Muir d226a973f7 core: upgrade to lucene 5 r1650327.
refactor _version docvalues migration to be more efficient.

closes #9206
2015-01-09 12:12:31 -05:00
Colin Goodheart-Smithe 91e00c6c8e Aggregations: Numeric metric aggregations are now formattable
You can now specify `format` in the request definition for most numeric metric aggregations. The exceptions are Percentile_Ranks, Cardinality and Value_Count as the response type of these can be different from the field type so the formatter won't work.

Closes #6812
2015-01-09 16:10:58 +00:00
Christoph Büscher 04cb09f44c [TEST] Add missing docs and tests for '_cat/segments'
The '_cat/segments' api was missing docs and a rest test which are added here.

Closes #5856
2015-01-09 12:29:11 +01:00
Simon Willnauer d2277d70ff [ENGINE] Simplify Engine construction and ref counting
Today the internal engine closes itself it the engine hits an exception
it can not recover from. This complicates a lot of refcounting issues
if such an exception happens during engine creation. This commit
only markes the engine as failed and let the user close it once the exception
bubbles up. Additionally it rolls back the indexwriter to prevent any changes after
the engine is failed.
2015-01-09 02:04:16 +01:00
Martijn van Groningen 592f517583 Serialize the rest status code, not the rest status enum. 2015-01-08 23:58:46 +01:00
David Pilato 6d58db8868 Mapping With a `null` Default Timestamp Causes NullPointerException on Merge
I have a field with a `null` [default `_timestamp` value](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-timestamp-field.html#mapping-timestamp-field-default) and when I try to update the mapping I get a server error caused by a `NullPointerException`

```
[2015-01-08 17:28:56,040][DEBUG][action.admin.indices.mapping.put] [...] failed to put mappings on indices [[feed_170_v1, feed_204_v1, feed_229_v1, feed_232_v1, feed_239_v1, feed_248_v1, feed_268_v1, feed_256_v1, feed_272_v1, feed_159_v1, feed_255_v1, feed_164_v1, feed_259_v1, feed_266_v1, feed_188_v1, feed_240_v1, feed_233_v1, feed_13_v1, feed_184_v1, feed_261_v1, feed_267_v1, feed_271_v1, feed_257_v1, feed_172_v1, feed_238_v1, feed_254_v1, feed_223_v1, feed_274_v1, feed_203_v1, feed_269_v1, feed_262_v1, feed_205_v1, feed_168_v1, feed_219_v1, feed_253_v1, feed_251_v1, feed_173_v1, feed_252_v1, feed_210_v1, feed_216_v1, feed_218_v1, feed_118_v1, feed_273_v1, feed_227_v1, feed_166_v1, feed_213_v1, feed_226_v1]], type [history]
java.lang.NullPointerException
        at org.elasticsearch.index.mapper.internal.TimestampFieldMapper.merge(TimestampFieldMapper.java:287)
        at org.elasticsearch.index.mapper.object.ObjectMapper.merge(ObjectMapper.java:936)
        at org.elasticsearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:693)
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$4.execute(MetaDataMappingService.java:508)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:329)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
```

https://github.com/elasticsearch/elasticsearch/blob/v1.4.2/src/main/java/org/elasticsearch/index/mapper/internal/TimestampFieldMapper.java#L286

Looks like the existence of default timestamp is not checked before use. The next line also has the same issue -- uses of default timestamp without checked to see if it's not null.

To reproduce:

```
$ curl -XPUT localhost:9200/twitter2

$ curl -XPUT localhost:9200/twitter2/tweet/_mapping -d '{
     "tweet" : {
         "_timestamp" : {
             "enabled" : true,
             "default" : null
         }
     }
}'

$ curl -XPUT localhost:9200/twitter2/tweet/_mapping -d '{
     "tweet" : {
         "_timestamp" : {
             "enabled" : true,
             "default" : null
         },
         "properties": {
             "user": {"type": "string"}
         }
     }
}'
```

Closes #9204.

(cherry picked from commit 62c6d63)
2015-01-08 21:33:17 +01:00
Ryan Ernst 7f9ffea97c Tests: Add upgrade step to static bwc tests 2015-01-08 11:53:48 -08:00
Ryan Ernst 060f963a8e Mappings: Remove allow_type_wrapper setting
Before Elasticsearch 1.0, the type was allowed to be passed as the root
element when uploading a document.  However, this was ambiguous if the
mappings also contained a field with the same name as the type.  The
behavior was changed in 1.0 to not allow this, but a setting was added
for backwards compatibility.  This change removes the setting for 2.0.
2015-01-08 09:13:40 -08:00
Martijn van Groningen ca4f27f40e Core: Added `_shards` header to all write responses.
The header indicates to how many shard copies (primary and replicas shards) a write was supposed to go to, to how many
shard copies to write succeeded and potentially captures shard failures if writing into a replica shard fails.

For async writes it also includes the number of shards a write is still pending.

Closes #7994
2015-01-08 18:10:08 +01:00
Ryan Ernst 1ad64a97ec Mappings: Remove includeExisting flag from adding ObjectMapper and
FieldMapper listeners

This flag is unused by the 2 places that add these listeners.
2015-01-08 09:08:54 -08:00
Simon Willnauer 959e3ca9da [CORE] Fold engine into IndexShard
This commit removes most of the Engine abstractions and removes
Engine exposure via dependency injection. It also removes the Holder
abstraction and makes the engine itself start at constrcution time.
It removes the start method from the engine entire which means no engine
instances exists until it's started. There is also no way to stop the
engine to restart, it needs to be an entire new Engine
2015-01-08 17:48:27 +01:00
Martijn van Groningen dedaf9387e Core: Also check if indices resolved via aliases resolution aren't closed and deal with this according to IndicesOptions.
Closes #9057
2015-01-08 16:45:34 +01:00
Martijn van Groningen b0b61ee0c3 Renamed allowNoIndices to failNoIndices and changed parameter order. 2015-01-08 16:43:56 +01:00
Simon Willnauer 78fc7c3f01 [TEST] Ensure shard lock is acquired before we try the timeout version 2015-01-08 15:37:31 +01:00
Colin Goodheart-Smithe ecfe72ebcc Indices API: Fix to make GET Index API consistent with docs
This fix ensures that calls to the GET alias/mappings/settings/warmers APIs return the aliases/mappings/settings/warmers object even if there is no content within them.. This make them consistent with the GET Index API docs and the breaking changes in 1.4 docs

Closes #9148
2015-01-08 08:48:44 +00:00
Boaz Leskes ad66d25fa2 Test: trace logging for testDeleteSafe 2015-01-07 23:16:19 +01:00
Simon Willnauer acf6132a99 Fix 1.5.0 Lucene version constant 2015-01-07 22:13:12 +01:00
Simon Willnauer 77493762e2 [TEST] Add back MockIndexEngine
This test class was lost accidentially in a8fa650
2015-01-07 16:38:21 +01:00
Adrien Grand fda727e20c Internal: assert that we do not call blocking code from transport threads.
This currently only adds checks to BaseFuture, but this should already cover
lots of client code. We could add more in the future, like interactions with
the filesystem and so on.

Close #9164
2015-01-07 14:08:40 +01:00
Martijn van Groningen 20f7be378b Removed parent parameter from update request, because it is just sets the routing.
The routing option should be used instead. The parent a child document points to can't be updated.

Closes #4538
2015-01-07 10:26:20 +01:00
Martijn van Groningen 687be70736 Made sure that named filters and queries defined in a wrapped query and filter are not lost.
Closes #6871
2015-01-07 09:10:16 +01:00
Martijn van Groningen c94d056454 Fixed a bug that was caused by specifying routing on a multi percolate request causing an ArrayIndexOutOfBoundsException.
The multi percolate shard responses are collected in an atomic array which uses the shard id is used as index, but the number of shards the multi percolate request was meant to go to was used as size of this array instead the total number of shards an index has. This caused the exception when routing was used.

Closes #6214
2015-01-07 08:49:25 +01:00
Simon Willnauer 7ec8973fbc [CORE] Delete shard content under lock
Once we delete the the index on a node we are closing all resources
and subsequently need to delete all shards contents from disk. Yet
this happens today under a lock (the shard lock) that needs to be
acquried in order to execute any operation on the shards data
path. We try to delete all the index meta-data once we acquired
all the shard lock but this operation can run into a timeout which causes
the index to remain on disk. Further, all shard data will be left on
disk if the timeout is reached.

This commit removes all the shards data just before the shard lock
is release as the last operation on a shard that belongs to a deleted
index.
2015-01-06 21:53:28 +01:00
Ryan Ernst f7f99b8dbf Stats: Added verbose option to segments api, with full ram tree as first
additional element per segment.

This commit adds a verbose flag to the _segments api.  Currently the
only additional information returned when set to true is the full
ram tree from lucene for each segment.
2015-01-06 10:04:52 -08:00
Simon Willnauer 09b4d0e097 [EXEC] Remove reduced stack size
We used to reduces the stack size to 256kb. This commit removes
this optimization and relies on the JVMs default.

Closes #9135
2015-01-06 17:35:45 +01:00
Adrien Grand bc86796592 Core: Remove terms filter cache.
This is our only cache which is not 'exact' and might allow for stalled results.
Additionally, a similar cache that we have and needs to perform lookups in other
indices in order to run queries is the script index, and for this index we rely
on the filesystem cache, so we should probably do the same with terms filters
lookups.

Close #9056
2015-01-06 17:21:20 +01:00