Commit Graph

1025 Commits

Author SHA1 Message Date
Areek Zillur 41d31541a6 Allow users to override the name for the rollover index 2016-06-09 13:43:19 -04:00
Areek Zillur 756af81b00 change rollover rest-api placeholder name 2016-06-09 12:53:24 -04:00
Areek Zillur a9f24ea2dc fail rollover request if rollover index already exists 2016-06-09 12:38:12 -04:00
Areek Zillur ae3eb15caa fix rest tests 2016-06-08 15:03:35 -04:00
Areek Zillur 76c59b7417 remove unused request param 2016-06-07 22:37:10 -04:00
Areek Zillur d96fe20e3a add named writable registry glue 2016-06-06 16:11:46 -04:00
Areek Zillur a30437f250 adds _rollover api 2016-06-03 16:09:24 -04:00
Martijn van Groningen 766789b0f0 ingest: added `ignore_failure` option to all processors
If this option is enabled on a processor it silently catches any processor related failure and continues executing the rest of the pipeline.

 Closes #18493
2016-06-01 10:29:12 +02:00
Mike McCandless 5c525e6606 Remove index_writer_max_memory stat from segment stats 2016-05-31 06:29:29 -04:00
Simon Willnauer 502a775a7c Add primitive to shrink an index into a single shard (#18270)
This adds a low level primitive operations to shrink an existing
index into a new index with a single shard. This primitive expects
all shards of the source index to allocated on a single node. Once the target index is initializing on the shrink node it takes a snapshot of the source index shards and copies all files into the target indices data folder. An [optimization](https://issues.apache.org/jira/browse/LUCENE-7300) coming in Lucene 6.1 will also allow for optional constant time copy if hard-links are supported by the filesystem. All mappings are merged into the new indexes metadata once the snapshots have been taken on the merge node.

To shrink an existing index all shards must be moved to a single node (one instance of each shard) and the index must be read-only:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_settings' -d '{
    "settings" : {
        "index.routing.allocation.require._name" : "shrink_node_name",
        "index.blocks.write" : true 
    }
}
```
once all shards are started on the shrink node. the new index can be created via:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_shrink/logs_single_shard' -d '{
    "settings" : {
        "index.codec" : "best_compression",
        "index.number_of_replicas" : 1
    }
}'
```

This API will perform all needed check before the new index is created and selects the shrink node based on the allocation of the source index. This call returns immediately, to monitor shrink progress the recovery API should be used since all copy operations are reflected in the recovery API with byte copy progress etc.

The shrink operation does not modify the source index, if a shrink operation should
be canceled or if the shrink failed, the target index can simply be deleted and
all resources are released.
2016-05-31 10:41:44 +02:00
Yannick Welsch dee34c916c Expand wildcards to closed indices in /_cat/indices (#18545)
Closed indices are already displayed when no indices are explicitly selected. This commit ensures that closed indices are also shown when wildcard filtering is used. It also addresses another issue that is caused by the fact that the cat action is based internally on 3 different cluster states (one when we query the cluster state to get all indices, one when we query cluster health, and one when we query indices stats). We currently fail the cat request when the user specifies a concrete index as parameter that does not exist. The implementation works as intended in that regard. It checks this not only for the first cluster state request, but also the subsequent indices stats one. This means that if the index is deleted before the cat action has queried the indices stats, it rightfully fails. In case the user provides wildcards (or no parameter at all), however, we fail the indices stats as we pass the resolved concrete indices to the indices stats request and fail to distinguish whether these indices have been resolved by wildcards or explicitly requested by the user. This means that if an index has been deleted before the indices stats request gets to execute, we fail the overall cat request. The fix is to let the indices stats request do the resolving again and not pass the concrete indices.

Closes #16419
Closes #17395
2016-05-25 10:02:14 +02:00
Tal Levy b40628d4e8 fix simulate spec test for new exception handling (#18564) 2016-05-24 22:32:50 -07:00
Tanguy Leroux 1f011f9dea Remove Delete-By-Query plugin
closes #18469
2016-05-24 13:28:20 +02:00
Martijn van Groningen 27cc2fe4dc Moved the percolator from core to its own module
Significant changes:
* AbstractQueryTestCase has moved to the test framework module, in order for query builder tests in modules and plugins
* Added support to AbstractQueryTestCase to register plugins
* Lift the restriction that only one percolator could be added per index. This validation existed in MapperService, but because the percolator moved to a module it could no longer exist there. Instead of bringing it back it was removed. This validation existed since the percolator cache only supported one percolator query per document, since the percolator cache has been removed this restriction could removed as well.
* While moving percolator tests to the new module, also removed a couple of tests for the deprecated percolate and mpercolate api. These APIs are now sugar  APIs for bwc and rediect to the searvh and msearvh APIs. Some tests were still testing as if percolate and mpercolate API did the percolation, but this no longer the case and these tests could be removed.
2016-05-24 11:01:57 +02:00
spalger be2ba53fca [rest api spec] fix doc urls 2016-05-20 13:48:16 -07:00
Spencer 332f6ffe59 [rest api spec] fix url for reindex api docs 2016-05-20 12:22:56 -07:00
Simon Willnauer 35e705877b Limit retries of failed allocations per index (#18467)
Today if a shard fails during initialization phase due to misconfiguration, broken disks,
missing analyzers, not installed plugins etc. elasticsaerch keeps on trying to initialize
or rather allocate that shard. Yet, in the worst case scenario this ends in an endless
allocation loop. To prevent this loop and all it's sideeffects like spamming log files over
and over again this commit adds an allocation decider that stops allocating a shard that
failed more than N times in a row to allocate. The number or retries can be configured via
`index.allocation.max_retry` and it's default is set to `5`. Once the setting is updated
shards with less failures than the number set per index will be allowed to allocate again.

Internally we maintain a counter on the UnassignedInfo that is reset to `0` once the shards
has been started.

Relates to #18417
2016-05-20 20:37:45 +02:00
Martijn van Groningen 80fee8666f percolator: Removed percolator cache
Before 5.0 for it was required that the percolator queries were cached in jvm heap as Lucene queries for two reasons:
1) Performance. The percolator evaluated all percolator queries all the time. There was no pre-selecting queries that are likely to match like we have today.
2) Updates made to percolator queries were visible in realtime, Today these changes are visible in near realtime. So updating no longer requires the percolator to have the queries in jvm heap.

So having the percolator queries in jvm heap via the percolator cache is now less attractive. Especially when there are many percolator queries then these queries can consume many GBs of jvm heap.
Removing the percolator cache does make the percolate query slower compared to how the execution time in 5.0.0-alpha1 and alpha2, but it is still faster compared to 2.x and before.
2016-05-20 14:52:16 +02:00
Tanguy Leroux a01ecb20ea Port Delete By Query to Reindex infrastructure
closes #16883
2016-05-19 16:07:50 +02:00
markharwood a846ff93e9 Aggregations fix: support include/exclude strings formatted for IP and date fields in terms and significant_terms aggregations.
Closes #17705
2016-05-18 16:21:55 +01:00
Tanguy Leroux d7a31c8cf7 Add missing builder.endObject() in FsInfo
closes #18433
2016-05-18 15:19:30 +02:00
Christoph Büscher de321fb159 Disabling nodes.stats/30_discovery.yaml rest test
This is tracked in issue #18433, so temporarily disabling
the tests.
2016-05-18 13:02:55 +02:00
polyfractal 978c1e3e36 [TEST] Add missing sort processor to test
Also fixes the naming of the sort REST test
to follow the numbering convention
2016-05-17 14:52:17 -04:00
Zachary Tong 7c46b57ff2 Add a Sort ingest processor
Sorts an array of values in ascending or descending order. If all elements are numerics, they will be sorted numerically. If values are strings, or mixtures of strings/numbers, the elements will be sorted lexicographically.
2016-05-17 12:06:48 -04:00
Chris Earle 9303ffbf82 Adding REST tests to ensure key_as_string behavior stays consistent 2016-05-13 11:48:49 -04:00
Adrien Grand 61b1f4ad0b Fix xcontent rendering of ip terms aggs. #18003
Currently terms on an ip address try to put their binary representation in the
json response. With this commit, they would return a formatted ip address:

```
      "buckets": [
        {
          "key": "192.168.1.7",
          "doc_count": 1
        }
      ]
```
2016-05-13 14:59:36 +02:00
Lee Hinman 1c54033e92 Merge branch 'pr/18068' 2016-05-10 08:27:43 -06:00
Jim Ferenczi aef78ceb13 Do not return fieldstats information for fields that exist in the mapping
but not in the index.
2016-05-09 19:24:03 +02:00
Tanguy Leroux 0ff5652fff Add node name to Cat Recovery
closes #8041
2016-05-06 16:59:53 +02:00
Nik Everett 230697c202 [reindex] Switch throttle to Float.POSITIVE_INFITINTY/"unlimited"
All other values are errors.

Add java test for throttling. We had a REST test but it only ran against
one node so it didn't catch serialization errors.

Add Simple round trip test for rethrottle request
2016-05-04 16:14:32 -04:00
Martijn van Groningen 7aca1389e2 ingest: Add `date_index_name` processor.
Closes #17814
2016-04-29 17:20:48 +02:00
David Pilato 2232a7cdf3 Merge branch 'pr/cat-size-time-units' 2016-04-29 15:09:14 +02:00
Jim Ferenczi 573c4f3ed1 Extend field stats:
* Add isSearchable and isAggregatable (collapsed to true if any of the instances of that field are searchable or aggregatable).
* Accept wildcards in field names.
* Add a section named conflicts for fields with the same name but with incompatible types (instead of throwing an exception).
2016-04-27 16:51:53 +02:00
Alexander Kazakov a8a33a1a94 Row-centric output for _cat/fielddata 2016-04-27 13:29:02 +03:00
Martijn Laarman 98a37a1e54 Remove trailing / in rest spec for ingest.simulate (#17976)
The url that takes an id has a trailing forward slash, not really an
error but as its the only url in the whole spec that does this it
triggered my OCD :)
2016-04-26 11:03:17 +02:00
Nik Everett 1c2e84ba46 Fail request if rescore window > 10,000
The setting is named `index.max_rescore_window` and defaults to
`index.max_result_window` which defaults to 10,000.
2016-04-22 11:10:01 -04:00
Martijn van Groningen c5ad2e2865 Changed indexed scripts to be stored in the cluster state instead of the `.scripts` index.
Also added max script size soft limit for stored scripts.

Closes #16651
2016-04-22 13:42:55 +02:00
Martijn van Groningen dd2184ab25 ingest: Streamline option naming for several processors:
* `rename` processor, renamed `to` to `target_field`
* `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats`
* `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties`
* `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties`

Closes #17835
2016-04-21 13:40:43 +02:00
Jun Ohtani 9eb242a5fe Analyze API : Rename filters/token_filters/char_filter to filter/token_filter/char_filter
Closes #15189
2016-04-21 18:05:11 +09:00
Martijn van Groningen 40c22fc654 percolator: removed .percolator type instead a field of type `percolator` should be configured before indexing percolator queries
* Added an extra `field` parameter to the `percolator` query to indicate what percolator field should be used. This must be an existing field in the mapping of type `percolator`.
* The `.percolator` type is now forbidden. (just like any type that starts with a `.`)

This only applies for new indices created on 5.0 and later. Indices created on previous versions the .percolator type is still allowed to exist.
The new `percolator` field type isn't active in such indices and the `PercolatorQueryCache` knows how to load queries from these legacy indices.
The `PercolatorQueryBuilder` will not enforce that the `field` parameter is of type `percolator`.
2016-04-19 11:20:31 +02:00
David Pilato 5e1f26c22a Add support for documented byte/size units and for micros as a time unit in _cat API
We advertise in our documentation that byte units are like `kb`, `mb`... But we actually only support the simple notation `k` or `m`.
This commit adds support for the documented form and keeps the non documented options to avoid any breaking change.

It also adds support for `micros`, `nanos` and `d` as a time unit in `_cat` API.

Remove the support for `b` as a SizeValue unit. Actually, for numbers, when using raw numbers without unit, there is no text to add/parse after the number. For example, you don't write `10` as `10b`. We support option like `size=` in `_cat` API which means that we want to display raw data without unit (singles).

Documentation updated accordingly.

Add test for the empty size option.

Fix missing TimeValues options for some cat APIs
2016-04-15 20:55:41 +02:00
jaymode 293ede3237 Enable installing the rest-api-spec artifact
Running `gradle install` on the rest-api-spec fails because there is no available install
task. This change applies the nexus plugin so that we can install and should also enable
publishing as part of the uploadArchives task.
2016-04-13 11:26:47 -04:00
Jun Ohtani 048d273408 Cat: cat health supports ts=0 option
If ts=0, cat health disable epoch and timestamp
Be Constant String timestamp and epoch
Move timestamp and epoch to Table
Add rest-api test and test

Closes #10109
2016-04-13 18:08:30 +09:00
Nik Everett 7908759949 Make aggregation registration more like query registration
Creates a class to hold behavior common to these `ParseFieldRegistry`s.

Also renames some weirdly named methods on StreamInput.
2016-04-12 14:50:44 -04:00
Igor Motov 81c59cae18 Add _cat/tasks
Adds new _cat endpoint that lists all tasks
2016-04-07 09:28:21 -06:00
Jimmy Jones f157dae053 Disallow unquoted field names, fix testcases using unquoted JSON 2016-04-06 14:37:15 -06:00
Spencer 7037670aeb [REST API] set correct default value
The correct default value for the `expand_wildcards` parameter to `indices.get_alias` is `all` as of all
f4d75f0212
2016-04-05 09:19:21 -07:00
Mpdreamz dd0c99bb65 index is a required url part 2016-04-04 14:14:44 +02:00
Mpdreamz 2944869ab9 Document task id's as string in the rest spec 2016-04-01 11:04:15 +02:00
Martijn Laarman dfe2c0ff0a Remove deprecated indices.get_aliases
This has been deprecated since the first release of Elasticsearch 1.0
2016-03-31 14:46:53 +02:00