Commit Graph

1221 Commits

Author SHA1 Message Date
Spencer 31847c1e9d [REST API] use a block literal for request bodies 2016-02-20 12:55:23 -08:00
Spencer a859595dcd [REST API] use a block literal for request bodies 2016-02-20 12:53:39 -08:00
Adrien Grand 4f8895eae3 Add a text field.
This new field is intended to replace analyzed string fields.
2016-02-15 10:43:44 +01:00
Jason Tedor 3bbd1c129e Remove host from cat nodes API
As the host and ip fields are always equal by design, the host field in
the cat nodes API is redundant and should be removed.

Closes #16656
2016-02-14 09:21:32 -05:00
Nik Everett 821a20f582 Merge branch 'master' into feature/reindex 2016-02-11 17:41:05 -05:00
Nik Everett 18808b7576 Move reindex from a plugin to a module 2016-02-11 17:39:49 -05:00
Adrien Grand bc47c577d2 Add a new `keyword` field.
The `keyword` field is intended to replace `not_analyzed` string fields. It is
indexed and has doc values by default, and doesn't support enabling term
vectors.

Although it doesn't support setting an analyzer for now, there are plans for
it to support basic normalization in the future such as case folding.
2016-02-11 18:19:53 +01:00
Igor Motov 99a7d8e41f Add task cancellation mechanism
Only tasks that extend CancellableTask can be cancelled using this mechanism. If a cancellable task has children it can elect to cancel all child tasks as well. In this case a special ban parent request is sent to all nodes. This request does two things: 1) it prevents any tasks with the banned parent task from being started, and 2) it cancels all currently running tasks that have the banned task as a parent. The ban is lifted as soon as the coordinating node notifies all other nodes that the cancelled task has finished executing. If the coordinating node leaves the cluster before it has a chance to lift its bans, all bans set by this coordinating node are automatically removed.

As an option a task can elect to automatically cancel all child tasks if their parent task was running on a node that just left the cluster. This option makes sense for cancellable heavy tasks that have no side-effects and only return results to the coordinating node. With the coordinating node gone, it doesn't make sense to run such tasks any longer since their results will be most likely discarded.
2016-02-09 22:30:57 -05:00
Yannick Welsch 0d11443aba Fix filters and null parameters in _aliases command
Closes #16549
Closes #16547
2016-02-09 21:43:42 +01:00
Andrej Kazakov 7f2b369dfd Use Accept header field in cat API
The cat API previously used the Content-Type header field for
determining the media type of the response. This is in opposition to the
HTTP spec which specifies the Accept header field for this purpose. This
commit replaces the use of the Content-Type header field with the Accept
header field in the cat API.

Closes #14421
2016-02-05 06:28:39 -05:00
Martijn van Groningen 7a6adfd93a ingest: Added foreach processor.
This processor is useful when all elements of a json array need to be processed in the same way.
This avoids that a processor needs to be defined for each element in an array.
Also it is very likely that it is unknown how many elements are inside an json array.
2016-02-04 23:44:01 +01:00
Simon Willnauer 450ee70038 Remove DFS support from TermVector API
Retrieving distributed DF for TermVectors is beside it's esotheric justification
a very slow process and can cause serious load on the cluster. We also don't have nearly
enough testing for this stuff and given the complexity we should remove it rather than carrying it
around.
2016-02-04 16:20:24 +01:00
Yannick Welsch 4937531a17 Remove obsolete version in ShardRouting
Closes #16243
2016-02-04 15:50:25 +01:00
Tal Levy 9e7e2ab10b remove DeDotProcessor from Ingest 2016-02-02 14:16:01 -08:00
Tal Levy 3191fc7347 Merge pull request #16355 from talevy/fix_ingest_exception
revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation
2016-02-02 14:11:24 -08:00
Tal Levy 0a1580eefa revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation 2016-02-02 14:08:22 -08:00
Greg Marzouka e7fc98a33f Remove detect_noop from REST spec
Unless this should be supported as a query string parameter instead, right now it only works when specified in the body.
2016-02-02 15:32:14 -05:00
Tal Levy fca442f4d1 Introduce Pipeline Factory Error Responses in Node Ingest
When there is an exception thrown during pipeline creation within
Rest calls (in put pipeline, and simulate) We now return a structured
error response to the user with details around which processor's
configuration is the cause of the issue, or which configuration property
is misconfigured, etc.
2016-01-29 13:37:27 -08:00
Jim Ferenczi 1343d6cbd1 Remove search_after from the query string param of the rest api spec.
Handle null values in search_after.
Ensure that the cluster is green after each index creation in the integ tests.
2016-01-27 19:21:01 +01:00
javanna 8006e5cd15 [TEST] re-enable and merge cluster settings REST tests
We used to have a disabled test around cluster put settings as it left cluster settings behind without a way to remove them. That has been in fixed in the cluster put settings api, so the test can be re-enabled.
2016-01-27 17:37:42 +01:00
Jim Ferenczi aea7660e37 Add search_after parameter in the Search API.
The search_after parameter provides a way to efficiently paginate from one page to the next. This parameter accepts an array of sort values, those values are then used by the searcher to sort the top hits from the first document that is greater to the sort values.
This parameter must be used in conjunction with the sort parameter, it must contain exactly the same number of values than the number of fields to sort on.

NOTE: A field with one unique value per document should be used as the last element of the sort specification. Otherwise the sort order for documents that have the same sort values would be undefined. The recommended way is to use the field `_uuid` which is certain to contain one unique value for each document.

Fixes #8192
2016-01-27 09:42:58 +01:00
Tal Levy ff0e8272cb [ingest] update test to verify that documents are deep-copied between verbose results 2016-01-26 14:12:42 -08:00
Martijn van Groningen df0be87b18 Merge pull request #16049 from elastic/feature/ingest
Merge feature/ingest branch into master branch.

This adds the ingest feature to ES that allows to preprocess document before indexing on an ingest node.
By default a node is an ingest node. Documents are preprocessed via a pipeline. A pipeline consists
out of one or more processors Each processor makes one or more modifications to a document processed.
There are many types of processors available out-of-the-box that are designed to make a specific change to a document being processed. In a cluster many pipeline can be configured via dedicated pipeline APIs. An new option on the bulk
and index APIs allows to control what pipeline is picked for preprocessing. If no pipeline is specified then the ingest
feature is skipped and no preprocessing takes place.
2016-01-26 13:41:13 +01:00
Martijn van Groningen 8b02f214c4 percolator: The percolate api shouldn't add field mappings for unmapped fields inside the document being percolated to the mapping.
Closes #15751
2016-01-26 10:26:46 +01:00
javanna 36d98478bf Merge branch 'master' into feature/ingest 2016-01-25 18:01:09 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Tal Levy 3a6c2d008e rename processor_tag to tag 2016-01-21 09:05:42 -08:00
Martijn van Groningen 602a0f183e Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-19 22:01:38 +01:00
Tal Levy 4ef85eda36 add default separator test to dedot rest test 2016-01-18 09:25:36 -08:00
Simon Willnauer 9562fb76bc expose default settings via rest API 2016-01-18 12:48:47 +01:00
Simon Willnauer 13e5547537 Add REST tests for reset index settings and for listing defaults. 2016-01-18 10:02:37 +01:00
Simon Willnauer dc05669fd9 replace unsupported setting translog.disable_flush with a high value of translog.flush_threshold_size 2016-01-18 09:23:35 +01:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Tal Levy 9f48df9736 Add on_failure support for verbose _simulate execution and introduce optional processor_tag to Processors 2016-01-15 14:56:20 -08:00
Tal Levy 1754eece66 introduce DeDotProcessor
fixes #15944.
2016-01-15 11:35:18 -08:00
javanna 9c06736dbd Merge branch 'master' into feature/ingest 2016-01-15 10:11:56 +01:00
javanna 07a82d0c09 make get alias expand to open and closed indices by default
This change affects get alias, get aliases as well as cat aliases. They all return closed indices too by default. get alias and get aliases also allow to return open indices only through the `expand_wildcards` option (set it to `open`).

Closes #14982
2016-01-14 10:40:31 +01:00
Martijn van Groningen f3883343cb Move the pipeline configuration from the dedicated index to the cluster state.
Closes #15842
2016-01-13 22:59:36 +01:00
javanna ea8065aa3d Merge branch 'master' into feature/ingest 2016-01-12 18:28:42 +01:00
Jason Tedor 1de2081ed3 Reintroduce five-minute and fifteen-minute load averages on Linux
This commit reintroduces the five-minute and fifteen-minute load stats
on Linux, and changes the format of the load_average field back to an
array.
2016-01-11 23:42:47 -05:00
javanna 90743d8db0 add REST test for bulk api integration with ingest 2016-01-11 19:04:34 +01:00
javanna ae69d46f92 move processors that have no deps to core, also move to core rest spec and tests and set node.inget to true by default 2016-01-08 10:39:39 +01:00
Adrien Grand 67d233cecd Remove warmers and the warmer API.
Warmers are now barely useful and will be removed in 3.0. Note that this only
removes the warmer API and query-based warmers. We still have warmers internally
for eg. global ordinals.

Close #15607
2016-01-07 09:57:07 +01:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Adrien Grand 6d3c9b074c Remove support for the `multi_field` type.
It is officially unsupported since version 1.0.
2015-12-30 12:03:15 +01:00
Lee Hinman 482843e27b Fix build to run correctly on FreeBSD
This adds the required changes/checks so that the build can run on
FreeBSD.

There are a few things that differ between FreeBSD and Linux:

- CPU probes return -1 for CPU usage
- `hot_threads` cannot be supported on FreeBSD

From OpenJDK's `os_bsd.cpp`:

```c++
bool os::is_thread_cpu_time_supported() {
  #ifdef __APPLE__
  return true;
  #else
  return false;
  #endif
}
```

So this API now returns (for each FreeBSD node):

```
curl -s localhost:9200/_nodes/hot_threads
::: {Devil Hunter Gabriel}{q8OJnKCcQS6EB9fygU4R4g}{127.0.0.1}{127.0.0.1:9300}
   hot_threads is not supported on FreeBSD
```

- multicast fails in native `join` method - known bug:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Which causes:

```
1> Caused by: java.net.SocketException: Invalid argument
1>    at java.net.PlainDatagramSocketImpl.join(Native Method)
1>    at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:179)
1>    at java.net.MulticastSocket.joinGroup(MulticastSocket.java:323)
1>    at org.elasticsearch.plugin.discovery.multicast.MulticastChannel$Plain.buildMulticastSocket(MulticastChannel.java:309)
```

So these tests are skipped on FreeBSD.

Resolves #15562
2015-12-22 12:36:04 -07:00
Simon Willnauer 6ea266a89c Merge branch 'master' into settings_prototype 2015-12-15 16:33:01 +01:00
Jun Ohtani fab44398d9 Analysis: Add detail response support
add explain option
fix char_filter bug

Closes #11076 #15257
2015-12-10 23:10:51 +09:00
Robert Muir e454fadc22 Merge branch 'master' into shave_mustache 2015-12-10 07:58:24 -05:00
Yannick Welsch bef0bedba9 Add support to _aliases endpoint to specify multiple indices and aliases in one action
Closes #15305
2015-12-09 19:08:27 +01:00
Robert Muir a6e1655fe9 fix integ tests 2015-12-09 00:30:32 -05:00
Jim Ferenczi 23aeaa88b2 Fixes random failures of org.apache.elasticsearch.test.rest.RestIT
RestTable: ignores right padding for the last cell of a column.
2015-12-08 20:52:24 +01:00
Myll 73a3c326c9 _cat APIs: remove space at the end of a line
Fixes #9464
2015-12-08 15:03:59 +01:00
Simon Willnauer 8502926327 fuck you linefeed 2015-12-08 14:39:16 +01:00
Simon Willnauer 2e27ee393f add rest API to reset settings 2015-12-08 14:39:16 +01:00
Lee Hinman f709b7283f Remove `GET` option for /_forcemerge
POST should be used to indicate this is not just a retrieval operation.

Resolves #15165
2015-12-03 13:56:15 -07:00
Jim Ferenczi e182072b6f Merge pull request #15017 from jimferenczi/fields_option
Refuse to load fields from _source when using the `fields` option and support wildcards.
2015-11-30 11:01:21 +01:00
Jim Ferenczi 731833cfc6 Fixes #14489
Do not to load fields from _source when using the `fields` option.
  Non stored (non existing) fields are ignored by the fields visitor when using the `fields` option.

Fixes #10783
  Support * wildcard to retrieve stored fields when using the `fields` option.
  Supported pattern styles are "xxx*", "*xxx", "*xxx*" and "xxx*yyy".
2015-11-30 11:00:32 +01:00
Clinton Gormley 27dac8dc2c REST spec: Added the verbose flag to indices.segments
Relates to #9111
2015-11-30 07:41:29 +01:00
Jayson Minard 815c53e6b4 body attribute was at wrong nesting level 2015-11-26 14:34:02 -03:00
Lee Hinman a25b407aeb Add support for headers in REST tests
This adds support for arbitrary headers sent with each REST request, it
will allow us to test things like different xcontent-encoding (see
50_with_headers.yaml for what this looks like).

Headers are specified at the same level as `catch`, so a request would
look like:

```yaml
- do:
    headers:
      Content-Type: application/yaml
    get:
      index: test_1
      type:  _all
      id:    1
```
2015-11-24 08:25:02 -07:00
Martijn van Groningen 48771f1a76 field stats: Added `min_value_as_string` and `max_value_as_string` response elements for all number based fields. The existing `min_value` and `max_value` will return the values as numbers instead.
Closes #14404
2015-11-23 08:48:28 +01:00
Xu Zhang 2e6d72de27 Catch exception when reading corrupted snapshot.
Single corrupted snapshot file shouldn't prevent listing all other
snapshot in repository.
2015-11-18 21:43:46 -08:00
Jason Tedor 185027a0ff Update REST tests to reflect changes to cat nodes default response
This commit updates the cat nodes REST test to include the CPU percent
that was recently added to the default output of the cat nodes response.
2015-11-17 14:51:03 -05:00
Jason Tedor 95c4846e58 Fix race condition in cat shards test
This commit fixes a test bug in the cat shards REST test. In
particular, there was a race condition in the test that would cause the
test to sometimes fail. The race condition is that some of the shards
would go to state STARTED after the sync flush was issued. These shards
would (correctly) show up in the output as having state started but
without a sync_id. However, the expected output was written to only
look for shards that have state STARTED and a sync_id, or shards that
are still INITIALIZING or are UNASSIGNED and (of course) do not have a
sync_id. The best approach here is to just simplify the test.
2015-11-13 12:16:22 -05:00
Jason Tedor 99abb76c78 Fix cat shards test bug 2015-11-13 09:31:44 -05:00
Jason Tedor a9ab35a487 Add sync_id to cat shards API
This commit adds the ability to get the sync_id from the cat shards API.

Closes #14705
2015-11-13 05:13:08 -05:00
Areek Zillur dd1c687ace Completion Suggester V2
The completion suggester provides auto-complete/search-as-you-type functionality.
This is a navigational feature to guide users to relevant results as they are typing, improving search precision.
It is not meant for spell correction or did-you-mean functionality like the term or phrase suggesters.

The completions are indexed as a weighted FST (finite state transducer) to provide fast Top N prefix-based
searches suitable for serving relevant results as a user types.

closes #10746
2015-11-07 17:46:27 -05:00
Yannick Welsch 825d0c64e6 Add duration field to /_cat/snapshots
Closes #14385
2015-11-04 10:34:00 +01:00
javanna b56bbf62dd Validate query api: move query parsing on the coordinating node
Similarly to what we did with the search api, we can now also move query parsing on the coordinating node for the validate query api. Given that the explain api is a single shard operation (compared to search which is instead a broadcast operation), this doesn't change a lot in how the api works internally. The main benefit is that we can simplify the java api by requiring a structured query object to be provided rather than a bytes array that will get parsed on the data node. Previously if you specified a QueryBuilder it would be serialized in json format and would get reparsed on the data node, while now it doesn't go through parsing anymore (as expected), given that after the query-refactoring we are able to properly stream queries natively. Note that the WrapperQueryBuilder can be used from the java api to provide a query as a string, in that case the actual parsing of the inner query will happen on the data node.

Relates to #10217
Closes #14384
2015-11-02 11:21:20 +01:00
Ryan Ernst 542522531a Build: Remove maven pom files and supporting ant files
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See #13930
2015-10-29 23:53:49 -07:00
Lee Hinman 3b5058017e Merge branch 'remove-optimize-rest' 2015-10-29 15:18:03 -06:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
xuzha 97ecd7bf5a Expose pending cluster state queue size in node stats
Add 3 stats about the queue: total queue size, number of committed cluster
states, and number of pending cluster states.
2015-10-28 10:59:15 -07:00
Lee Hinman 3a458af0b7 Remove /_optimize REST API endpoint
The `/_optimize` endpoint was deprecated in 2.1.0 and can now be removed
entirely.
2015-10-27 10:17:16 -06:00
javanna dc900a08a6 Remove "query" query and fix related parsing bugs
We have two types of parse methods for queries: one for the inner query, to be used once the parser is positioned within the query element, and one for the whole query source, including the query element that wraps the actual query.

With the search refactoring we ended up using the former in count, cat count and delete by query, whereas we should have used the former.  It ends up working properly given that we have a registered (deprecated) query called "query", which used to allow to wrap a filter into a query, but this has the following downsides:
1) prevents us from removing the deprecated "query" query
2) we end up supporting a top level query that is not wrapped within a query element (pre 1.0 syntax iirc that shouldn't be supported anymore)

This commit finally removes the "query" query and fixes the related parsing bugs. We also had some tests that were providing queries in the wrong format, those have been fixed too.

Closes #13326
Closes #14304
2015-10-27 14:54:30 +01:00
Yannick Welsch 5959058719 Unique repository names for rest tests 2015-10-26 21:02:25 +01:00
Yannick Welsch 2f10300a03 Merge pull request #14247 from ywelsch/feature/cat-snapshots
Add cat API for repositories and snapshots
2015-10-26 18:40:39 +01:00
Yannick Welsch ca75b7b6ce Add cat API for repositories and snapshots
Closes #14247
Closes #13919
2015-10-26 18:37:10 +01:00
javanna 75cedca0da Remove search exists api
Closes #13682
Closes #13911
2015-10-21 17:39:32 +02:00
Lee Hinman 9ea4909035 Add Force Merge API, deprecate Optimize API
This adds an API for force merging lucene segments. The `/_optimize` API is now
deprecated and replaced by the `/_forcemerge` API, which has all the same flags
and action, just a different name.
2015-10-20 09:00:24 -06:00
Spencer ff9999876c [REST tests] prevent backtracking in cat.nodeattrs 2015-10-19 09:36:19 -07:00
Colin Goodheart-Smithe 4557d1b560 Merge branch 'master' into feature/search-request-refactoring
# Conflicts:
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/FunctionScoreTests.java
2015-10-08 11:39:34 +01:00
Igor Motov a358f34276 Expose nodes operation timeout in REST API
Currently it's not possible to specify a timeout for nodes operations (such as node info, node stats, cluster stats and hot threads) via REST-based APIs.
2015-10-07 18:07:59 -04:00
Robert 331d2d9955 Update update.json
Missing spec for [detect_noop](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html?q=update%20a#_literal_detect_noop_literal) parameter.
2015-10-07 16:13:31 -04:00
Boaz Leskes bcb3fab6ac Engine: Remove Engine.Create
The `_create` API is handy way to specify an index operation should only be done if the document doesn't exist. This is currently implemented in explicit code paths all the way down to the engine. However, conceptually this is no different than any other versioned operation - instead of requiring a document is on a specific version, we require it to be deleted (or non-existent). This PR removes Engine.Create in favor of a slight extension in the VersionType logic.

There are however a couple of side effects:
- DocumentAlreadyExistsException is removed and VersionConflictException is used instead (with an improved error message)
- Update will reject version parameters if the upsert option is used (it doesn't compute anyway).
- Translog.Create is also removed infavor of Translog.Index (that's OK because their binary format was the same, so we can just read Translog.Index of the translog file)

Closes #13955
2015-10-07 12:37:34 +02:00
Greg Marzouka 9200c446a1 Merge pull request #13946 from elastic/fix/indices-get-restspec
Add options for indices.get feature
2015-10-06 14:28:11 -04:00
javanna 1915c74e93 Merge branch 'master' into feature/search-request-refactoring 2015-10-06 16:21:58 +02:00
javanna f89de33548 Merge branch 'master' into feature/search-request-refactoring 2015-10-06 14:28:31 +02:00
Martijn Laarman fd1cd60e78 Fix indices.get_field_mapping rest tests
After `field` has been renamed to `fields` in #13902
2015-10-06 12:40:08 +02:00
Martijn Laarman a0fa83e3df Merge pull request #13678 from elastic/fix/cluster-health-api
In cluster health REST spec, {index} can be one or many indices and should be typed to list.
2015-10-06 11:28:47 +02:00
Martijn Laarman 47871eb854 Merge pull request #13711 from elastic/fix/api-delete-search-template
id route value is required
2015-10-06 11:28:35 +02:00
Martijn Laarman c12cb0b60a Merge pull request #13889 from elastic/fix/indices-get-template-restspec
indices get template accepts a list of names
2015-10-06 11:28:13 +02:00
Martijn Laarman 536ee2b5eb Merge pull request #13900 from elastic/fix/indices-open-restspec
indices.open takes a list of indices
2015-10-06 11:27:52 +02:00
Martijn Laarman fd6652f59f Merge pull request #13901 from elastic/fix/indices-close-restspec
indices.close takes a list of indices
2015-10-06 11:27:12 +02:00
Martijn Laarman e4049ccd8e Merge pull request #13902 from elastic/fix/get-field-mapping-restspec
Get field mapping documented fields as field
2015-10-06 11:26:46 +02:00
Honza Král f2fa279af1 [test] remove index= from catch: /regex/ in date math tests
This breaks on some clients and is not required for the actual test
2015-10-05 20:52:52 +02:00
Greg Marzouka eef34475fd Add options for indices.get feature
As described here https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html?q=get%20index#_filtering_index_information
2015-10-05 12:48:30 -04:00
Colin Goodheart-Smithe ec51e8e31d Merge branch 'master' into feature/search-request-refactoring 2015-10-05 13:31:52 +01:00
Martijn Laarman 6c7e135b49 Get field mapping documented fields as field 2015-10-02 12:54:07 +02:00
Martijn Laarman 80b5a58835 indices.close takes a list of indices
not single index
2015-10-02 12:30:40 +02:00
Martijn Laarman c3f25a0713 indices.open takes a list of indices
not single index
2015-10-02 12:28:53 +02:00
Martijn Laarman 56cbbc4710 indices get template accepts a list of names
not just string
2015-10-01 20:32:22 +02:00
André Carvalho 03c6e8e1cb Adds disk used by indices to _cat/allocation
Sets Store flag on request
2015-10-01 08:16:58 -03:00
Colin Goodheart-Smithe a21238beda Merge branch 'master' into feature/search-request-refactoring
# Conflicts:
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/ChildQuerySearchTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/IndexedScriptTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SearchFieldsTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SearchStatsTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SearchTimeoutTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SimpleSortTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/script/groovy/GroovyScriptTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/script/groovy/GroovySecurityTests.java
2015-09-29 14:40:27 +02:00
Robert Muir e0d42739dd Factor groovy out of core into lang-groovy 2015-09-28 20:17:45 -04:00
Colin Goodheart-Smithe 44722700b0 Merge branch 'feature/query-refactoring' into feature/search-request-refactoring
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/fetch/FetchSubPhasePluginIT.java
2015-09-25 13:04:50 +01:00
Martijn van Groningen 93ad696966 Index name expressions should not be broken up
Closes #13665
2015-09-25 11:28:52 +02:00
Colin Goodheart-Smithe 9f08d48d34 Merge branch 'feature/query-refactoring' into feature/search-request-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/index/query/IndexQueryParserService.java
#	core/src/test/java/org/elasticsearch/script/IndexedScriptIT.java
#	core/src/test/java/org/elasticsearch/script/OnDiskScriptIT.java
2015-09-25 10:14:53 +01:00
Colin Goodheart-Smithe 2dce527b81 rest test fixes (hopefully) 2015-09-23 21:15:13 +01:00
Colin Goodheart-Smithe 4996b07205 update rest tests to expect default boost to be returned in warmer responses 2015-09-23 21:02:55 +01:00
Simon Willnauer 6c335f4a47 Parse source, extra-source and template-source on the coordinating node
This commit removes all the opaque bytes for extra_source and template_source.
Instead source and extra_source etc. are represented as SearchSourceBuilder which can
in-place be modified and is updated with the content of the request parameters.
Template Source is parsed and evaluated which in-turn replaces the actual source.
2015-09-23 12:24:07 +02:00
Robert Muir 689af1a6d6 Factor expressions scripts out to lang-expression plugin 2015-09-22 20:33:47 -04:00
Martijn Laarman 2ee82fc513 id route value is required
id route value is required
2015-09-22 13:08:37 +02:00
Martijn Laarman fcd8606f98 {index} can be one or many indices and should be typed to list.
{index} can be one or many indices and should be typed to list.
2015-09-21 17:38:10 +02:00
Ryan Ernst 18c519145d Remove unnecessary copies of license and notice files
We moved a lot of repositories into elasticsearch, but in their new
location they retained their LICENSE.txt and NOTICE.txt files. These are
all the same, and having the license and notice and the root of the
repository should be sufficient.
2015-09-18 17:48:30 -07:00
Nik Everett 820b721788 [test] Rename start to verify_index in cat test
This was renamed in other places but not here.

Closes #13489
2015-09-10 16:58:28 -04:00
Nik Everett 56c3471851 Fix test for _cat/nodeattrs
Adds a node attribute to all test runs and uses the attribute to test
`_cat/nodeattrs`.

Note that its quite possible create an impressively slow regex while doing
this and you have to be careful. See comment in commit for more if curious.

Closes #12558
2015-09-10 10:50:51 -04:00
Nik Everett 3839d15ea0 Merge branch 'pr/13130' 2015-09-10 10:13:01 -04:00
Nik Everett 2574b5e7f4 [test] Fix help test for _cat/shards
We added a new field and the help test didn't expect it to be there.
2015-09-10 09:35:03 -04:00
Martijn Laarman 8e7be8b552 part types are not string but list on the _search_shards endpoint 2015-09-10 12:02:11 +02:00
Martijn Laarman aaf7116bfb synced flush incorrectly documented url parameters as path parameters 2015-09-10 12:00:06 +02:00
Adrien Grand 0c26e7cd83 Remove the scan and count search types.
These search types have been deprecated in 2.1 and 2.0 respectively, and will
be removed in 3.0.
2015-09-07 15:18:45 +02:00
Simon Willnauer 796701d52e Move version to 3.0.0-SNAPSHOT 2015-09-03 10:43:28 +02:00
Isabel Drost-Fromm 8cd86a615a Adds template support to _msearch resource
Much like we already do with search this adds templating support to the _msearch resource.

Closes #10885
2015-09-01 11:54:43 +02:00
Nik Everett c180defb10 [CAT] Default verbose to false
Closes #13156
2015-08-28 11:15:44 -04:00
Nik Everett 9eb684da51 Default detect_noop to true
detect_noop is pretty cheap and noop updates compartively expensive so this
feels like a sensible default.

Also had to do some testing and documentation around how _ttl works with
detect_noop.

Closes #11282
2015-08-27 10:34:18 -04:00
Clinton Gormley 33e04083a7 Corrected some typos in the REST spec 2015-08-27 11:28:27 +02:00
Clinton Gormley a254b2da29 REST spec: Added update_all_types to indices.put_mapping and indices.create
Closes #12840
2015-08-26 15:33:23 +02:00
Spencer 56d7cb8bd8 Update indices.shard_stores.json
Correct documentation url
2015-08-25 13:21:04 -07:00
David Pilato c189c1a1ec [maven] also rename artifactId for dev-tools and rest-api-spec 2015-08-18 13:41:12 +02:00
David Pilato 0014728a0d [maven] rename maven names for parent, dev-tools and rest spec
Follow up for previous commit:

```
[INFO] Build Tools and Resources .......................... SUCCESS [  0.158 s]
[INFO] Rest API Specification ............................. SUCCESS [  0.002 s]
[INFO] Elasticsearch Parent POM ........................... SUCCESS [  0.122 s]
```
2015-08-18 13:38:49 +02:00
Alexander Reelsen d1c93fb573 Release: Remove aws-maven plugin/improve release docs
In order to have consistent deploys across several repositories,
we should deploy to sonatype first, then mirror those contents,
and then upload to s3.

This means, the aws wagon is not needed anymore.
2015-08-17 15:39:22 +02:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Alexander Reelsen e869c87119 Build: Adding aws-maven extension to dev-tools and rest-api-spec 2015-08-14 17:30:00 +02:00
Simon Willnauer c1405d26b5 fixtypo 2015-08-14 15:27:23 +02:00
Simon Willnauer dd1b0d84a6 Also deploy top level artifacts to S3
This commit adds the S3 wagon release profile also to dev-tools
and rest-api-spec and makes the actual repository path / bucket
configurable.
2015-08-14 15:21:06 +02:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
Simon Willnauer fe3179d9cc Return `408 REQUEST_TIMEOUT` if `_cluster/health` times out
Today we return `200 OK` which is misleading since we really didn't
produce a valid response / didn't wait long enough.
2015-08-12 17:51:20 +02:00
Colin Goodheart-Smithe 6abc69c488 Packaging: Makes sure all POMs contain a description
Adds an explicit description the RPM package so it doesn't inherit the description from the POM.

Closes #12550

Also, modified descriptions for deb and rpm packages to be the same and to reference the documentation rather than listing features that are out of date.
2015-08-10 15:21:24 +01:00
Igor Motov f71c9a25a1 Check for incompatible mappings while upgrading old indices
Conflicting mappings that were allowed before v2.0 can cause runaway shard failures on upgrade. This commit adds a check that prevents a cluster from starting if it contains such indices as well as restoring such indices from a snapshot into already running cluster.

Closes #11857
2015-08-04 18:13:29 -06:00
Robert Muir 6f9a067197 Change master branch back to 2.0-beta1 2015-08-04 15:38:21 -04:00
Simon Willnauer 2b2657279d Use UTC instead of default timezone for creation date in CAT endpoint
this change was added recently which uses default timezone for the creation
date on CAT endpoints. We should be consistent and use UTC across the board.
This commit adds #getDefaultTimzone() to forbidden API and fixes the REST tests.

Relates to #11688
2015-08-04 14:38:23 +02:00
Simon Willnauer 6753f7f03e Cut over master to 2.0.0-SNAPSHOT 2015-08-04 10:54:12 +02:00
Nik Everett 2078100390 [TESTS] cat indices times for UTC - timezones 2015-08-03 11:46:27 -04:00
szroland 94f8b8f2cf remove extra space in test regexp 2015-08-02 21:52:00 +02:00
szroland 9bb904b155 testing output, both with default field names and shorthand aliases 2015-08-02 20:53:03 +02:00
Alexander Reelsen 291bac0f04 Tests: Silencing failing node attrs REST test 2015-07-30 17:48:01 +02:00
Dave Parfitt f209809716 Add _cat/nodeattrs API
This provides a _cat/nodeattrs API call, which presents
custom node attributes in a denormalized table.

Closes #8000
2015-07-29 16:11:18 -04:00
Martijn van Groningen a14913f7b6 Left over from the `query_cache` to `request_cache` rename. 2015-07-27 13:28:15 +02:00
Jason Tedor 8637784a2a Add scroll stats to cat API
Adds scroll stats at the node, shard and index levels to the cat API.

Closes #12330
2015-07-19 13:41:46 -04:00
Areek Zillur 99363bd35d [TEST] fix test bug 2015-07-16 18:57:04 -04:00
Areek Zillur 7a21d846bb A new `_shard_stores` API provides store information for shard copies of indices.
Store information reports on which nodes shard copies exist, the shard
copy version, indicating how recent they are, and any exceptions
encountered while opening the shard index or from earlier engine failure.

closes #10952
2015-07-16 18:28:07 -04:00
Clinton Gormley d4b94a9be2 Revert "REST: Fixed msearch and mpercolate structured exception tests"
This reverts commit f20b730eca.
2015-07-16 17:06:45 +02:00
Clinton Gormley f20b730eca REST: Fixed msearch and mpercolate structured exception tests 2015-07-16 15:27:07 +02:00
Simon Willnauer def08bd594 Render structured exceptions in mget / mpercolate
Instead of rendering only the exception message this commit
adds structured exception rendering to mget and mpercolate
2015-07-14 17:52:28 +02:00
Simon Willnauer 7db293c616 Generify Index and Shard exceptions
Today we have a intermediate hierarchy for shard and index exceptions
which makes it hard to introduce generic exceptions like ResourceNotFoundException
intoduced in this commit. This commit breaks up the hierarchy by adding index and shard
as a special internal header that gets rendered for every exception that fills that header.
This commit removes dedicated exceptions like `IndexMissingException` or
`IndexShardMissingException` in favour of `ResourceNotFoundException`
2015-07-14 16:31:49 +02:00
Simon Willnauer fbafd14f23 [TEST] Don't use replicas to stabelize assertion.
This test asserts that the first recovery result is of type SNAPSHOT.
This assertion might not be true depending on the rendering order if
a replica is recovered quick enough. This commit disables replicas and
their recovery since it's not the purpose of this test.
2015-07-14 10:32:20 +02:00
uboness b40186652c updated the elasticsearch versioning format
Moving to from `X.Y.Z.beta1`/`X.Y.Z.RC1` to `X.Y.Z-beta1`/`X.Y.Z-rc1`
2015-07-13 20:26:37 +02:00
Jack Conradson 82b112b911 Minor fix to a failing REST test due to the prior check-in. 2015-07-10 19:30:28 -07:00
Simon Willnauer e0708813a9 Make 2.0.0.beta1-SNAPSHOT the current version.
Today everything is tight to having the next version as the latest.
In order to work towards 2.0.0.beta1 we need to fix all the usage of
2.0.0-SNAPSHOT to reflect the version we will release soon.
Usually we do this on the release branch but to simplify things I wanna
keep this on master for now and move to 2.1.0-SNAPSHOT on master once
we created a 2.0 branch.

Closes #12148
2015-07-09 21:24:32 +02:00
Tanguy Leroux 26df78a90a Fix REST test for cat.nodes (Windows load average OS stats return -1.00) 2015-07-09 19:52:52 +02:00
Igor Motov f303a1d4eb Snapshot info should contain version of elasticsearch that created the snapshot
This information was stored with the snapshot but wasn't available on the interface. Knowing the version of elasticsearch that created the snapshot can be useful to determine the minimal version of the cluster that is required in order to restore this snapshot.

Closes #11980
2015-07-09 12:02:33 -04:00
Lee Hinman 2213a5aa81 Revert "[TEST] Add REST awaitsFix for bulk fields test"
This reverts commit 799e801980.
2015-07-08 12:00:37 -06:00
Lee Hinman 799e801980 [TEST] Add REST awaitsFix for bulk fields test
Relates to #12133
2015-07-08 11:59:34 -06:00
Adrien Grand ccbea2e5d2 Fix parsing of the `fields` parameter of bulk requests. 2015-07-08 19:55:24 +02:00
Jason Tedor b61709c716 Add support for retrieving fields in bulk updates
This commit adds support to retrieve fields when using the bulk update API. This functionality was previously available for the update API
but not for the bulk update API.

Closes #11527
2015-07-08 12:06:20 -04:00
Tanguy Leroux 30892c4129 Remove network stats & info 2015-07-07 21:16:42 +02:00
Igor Motov 177180ac94 Fix snapshot-related REST integration tests 2015-07-06 14:16:28 -04:00
David Pilato e7a6b51bab [maven] change groupId / artifactId
When we generate our project, we can get something like:

```
├── dev-tools
├── elasticsearch
├── elasticsearch-parent
├── elasticsearch-plugin
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   └── elasticsearch-lang-python
├── rest-api-spec
└── securemock
```

I propose here to use a common naming for artifacts: start always with `elasticsearch-`.
Also, move `elasticsearch-plugin` to `org.elasticsearch.plugin` groupId.

So we could have:

```
├── elasticsearch
├── elasticsearch-dev-tools
├── elasticsearch-parent
├── elasticsearch-rest-api-spec
├── elasticsearch-securemock
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   ├── elasticsearch-lang-python
│   └── elasticsearch-plugin
```
2015-07-06 17:17:07 +02:00
Ryan Ernst 410704fd18 fix rest tests to not depend on short name access of mappings 2015-07-02 11:37:18 -07:00
szroland 68d658a30f Recovery: Fix wrong reused file bytes in Recovery API reports
Simple snapshot.restore test case that also exposes the bug mentioned in #11876

Fix #11876
Closes #11965
2015-07-01 09:08:53 +02:00
Martijn van Groningen ef9d70b9b3 field stats: added index constraints
Field stats index constraints allows to omit all field stats for indices that don't match with the constraint. An index
constraint can exclude indices' field stats based on the `min_value` and `max_value` statistic. This option is only
useful if the `level` option is set to `indices`.

For example index constraints can be useful to find out the min and max value of a particular property of your data in
a time based scenario. The following request only returns field stats for the `answer_count` property for indices
holding questions created in the year 2014:

curl -XPOST 'http://localhost:9200/_field_stats?level=indices' -d '{
   "fields" : ["answer_count"] <1>
   "index_constraints" : { <2>
      "creation_date" : { <3>
         "min_value" : { <4>
            "gte" : "2014-01-01T00:00:00.000Z",
         },
         "max_value" : {
            "lt" : "2015-01-01T00:00:00.000Z"
         }
      }
   }
}'

Closes #11187
2015-07-01 08:47:03 +02:00
Colin Goodheart-Smithe a847dd267c [TEST] Fix location of render_search_template.json 2015-06-30 17:45:43 +01:00
Colin Goodheart-Smithe d9ab3cba77 Search Templates: Adds API endpoint to render search templates as a response
Closes #6821
2015-06-30 16:57:23 +01:00
szroland e2d76aaa20 remove _create from POST to match PUT path 2015-06-29 21:55:52 +02:00
Masaru Hasegawa 80d0f1e5ea Fix RecoveryState timestamps
Use System.currentTimeMillis() for wall clock time and use System.nanoTime() for delta.
Fixes #11870.
2015-06-29 19:24:23 +09:00
Adrien Grand 38f5cc236a Rename caches.
In order to be more consistent with what they do, the query cache has been
renamed to request cache and the filter cache has been renamed to query
cache.

A known issue is that package/logger names do no longer match settings names,
please speak up if you think this is an issue.

Here are the settings for which I kept backward compatibility. Note that they
are a bit different from what was discussed on #11569 but putting `cache` before
the name of what is cached has the benefit of making these settings consistent
with the fielddata cache whose size is configured by
`indices.fielddata.cache.size`:
 * index.cache.query.enable -> index.requests.cache.enable
 * indices.cache.query.size -> indices.requests.cache.size
 * indices.cache.filter.size -> indices.queries.cache.size

Close #11569
2015-06-29 10:15:27 +02:00
Simon Willnauer dc67bd0021 Render strucutred exception in multi search
MultiMatch still only returns the exception message but should return the
actual exception and render it in a structured fashion
2015-06-24 16:21:41 +02:00
Simon Willnauer fcdcce3bba Consolidate shard level abstractions
This commit consolidates several abstractions on the shard level in
ordinary classes not managed by the shard level guice injector.

Several classes have been collapsed into IndexShard and IndexShardGatewayService
was cleaned up to be more lightweight and self-contained. It has also been moved into
the index.shard package and it's operation is renamed from recovery from "gateway" to recovery
from "store" or "shard_store".

Closes #11847
2015-06-24 15:18:04 +02:00
Ryan Ernst 12e7cbe92b Mappings: Lockdown _timestamp
This is a follow up to #8143 and #6730 for _timestamp. It removes
support for `path`, as well as any field type settings, and
enables docvalues for _timestamp, for 2.0.  Users who need to
adjust these settings can use a date field.
2015-06-22 10:21:03 -07:00
Alexander Reelsen 88f8d58c8b Cluster Health: Add max wait time for pending task and active shard percentage
In order to get a quick overview using by simply checking the cluster state
and its corresponding cat API, the following two attributes have been added
to the cluster health response:

* task max waiting time, the time value of the first task of the
  queue and how long it has been waiting
* active shards percent: The percentage of the number of shards that are in
  initializing state

This makes the cluster health API handy to check, when a fully restarted
cluster is back up and running.

Closes #10805
2015-06-22 15:04:47 +02:00
Colin Goodheart-Smithe 772d0cc6e7 Build: Make rest-spec-api a project so eclipse build works
The change makes rest-spec-api a project in the same way as we build dev-tools. it packages the tests and api in a bundle using the maven-remote-resources-plugin and uses the same plugin in the plugins and core pom to unpack the rest-api-spec into the target directory and references the rest tests there in the test resources.

The main stimulus for this change is that for those using Eclipse the current build does not work. After running `mvn eclipse:eclipse` the Eclipse IDE errors because the rest-api-spec is outside of the project scope, meaning that every time the command is run (required whenever any dependencies change), the class path of all the projects has to be manually fixed.
2015-06-22 11:41:44 +01:00
Igor Motov 36da42c93b Show human readable Elasticsearch version that created index and date when index was created
By setting human parameter to true, it's now possible to see human readable versions of Elasticsearch that created and updated the index as well as the date when the index was created.

Closes #11484
2015-06-17 18:52:01 -04:00
Alex Ksikes 3f6dae1a73 More Like This: renamed `ignore_like` to `unlike`
This changes the parameter name `ignore_like` to the more user friendly name
`unlike`. This later feature generates a query from the terms in `A` but not
from the terms in `B`. This translates to a result set which is like `A` but
unlike `B`. We could have further negatively boosted any documents that have
some `B`, but these documents already do not receive any contribution from
having `B`, and would therefore negatively compete with documents having `A`.

Closes #11117
2015-06-17 17:18:50 -05:00
Shay Banon 4e0d65453f Add Unassigned meta data
Unassigned meta includes additional information as to why a shard is unassigned, this is especially handy when a shard moves to unassigned due to node leaving or shard failure.

The additional data is provided as part of the cluster state, and as part of `_cat/shards` API.

The additional meta includes the timestamp that the shard has moved to unassigned, allowing us in the future to build functionality such as delay allocation due to node leaving until a copy of the shard is found.
closes #11653
2015-06-16 17:01:48 +02:00
Boaz Leskes e559eafa19 Revert "Test: fix removal of e validate_query rewrite YAML test"
This reverts commit e461852548.
2015-06-15 21:57:05 +02:00
Boaz Leskes e461852548 Test: fix removal of e validate_query rewrite YAML test 2015-06-15 20:29:06 +02:00
gmarz 3b93653863 [TEST] Remove validate_query rewrite YAML test 2015-06-15 12:52:06 -04:00
gmarz 8dd7dae5b9 [SPEC] Add `rewrite` query parameter to the `indices.validate_query` API 2015-06-15 11:13:46 -04:00
Simon Willnauer 43ce27d3bd [TEST] Shards can also be RELOCATING 2015-06-15 10:07:40 +02:00
Honza Král 09b5f90779 [API] correct link for flush_synced docs 2015-06-09 21:29:02 +02:00
Nirmal Chidambaram 931b9f9c74 Filtered out non data-nodes in relevant cat api
Closes #9214
Closes #9287
2015-06-08 16:05:42 +02:00
Boaz Leskes e359698605 test: reduce replicas for indices synced flush test
We have to make sure all shards are started to know the synced flush will hit them all. Shards that are still initializing during the sync flush may be missed and confuse the stats call
2015-06-06 21:10:25 +02:00
Adrien Grand 7c698146f5 Rest: Add all meta fields to the top level json document.
Some of our meta fields (such as _id, _version, ...) are returned as top-level
properties of the json document, while other properties (_timestamp, _routing,
...) are returned under `fields`. This commit makes all meta fields returned
as top-level properties.

So eg. `GET test/test/1?fields=_timestamp,foo` would now return

```json
{
   "_index": "test",
   "_type": "test",
   "_id": "1",
   "_version": 1,
   "_timestamp": 10000000,
   "found": true,
   "fields": {
     "foo": [ "bar" ]
   }
}
```

while it used to return

```json
{
   "_index": "test",
   "_type": "test",
   "_id": "1",
   "_version": 1,
   "found": true,
   "fields": {
     "_timestamp": 10000000,
     "foo": [ "bar" ]
   }
}
```
2015-06-04 23:42:17 +02:00
Michael McCandless e1197dfea9 Merge branch 'master' into require_units
Conflicts:
	src/main/java/org/elasticsearch/action/bulk/BulkRequest.java
	src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexUpgradeService.java
	src/main/java/org/elasticsearch/node/internal/InternalSettingsPreparer.java
	src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java
2015-06-04 13:59:10 -04:00
Britta Weber d23449ee85 [test] fix name in synced flush test 2015-05-29 13:48:37 +02:00
Clinton Gormley 18c8394fbd REST spec: Renamed indices.flush.synced to indices.flush_synced 2015-05-29 12:30:43 +02:00
Britta Weber 87a0c76e9c Merge remote-tracking branch 'boaz/index_seal_to_flush_sync' 2015-05-29 10:31:03 +02:00
Igor Motov c695f35bca Tests: make REST upgrade test more reliable
Make sure that all shards are started to avoid flush conflicts.
2015-05-28 12:37:52 -10:00
Igor Motov 6980286ba4 Core: fix upgrade response serialization 2015-05-28 09:46:46 -10:00
Alexander Reelsen 38639074b4 Testing: Ensure cat API REST tests are unaffected by randomization
The wildcard cat API REST tests relied on bulk.max and bulk.min in
the thread_pool response. However due to the thread pool types being
randomized in InternalTestCluster, the min/max values were not guaranteed
to exist (the cached thread pool type is unbounded and thus does not have a
max value).

In order to prevent this, the test has been removed and now the cat
nodes test is used for wildcard testing, which always returns stats
about the heap.
2015-05-28 16:24:41 +02:00
Clinton Gormley cbb7b633f6 REST tests: Fixed bad YAML in search/10_source_filtering 2015-05-27 19:35:00 +02:00
Alexander Reelsen 9d5e789508 Cat API: Do not rely on hashmap for sorted entries
The tests for the recently added added wildcard feature were
relying on order of the hashmap being used, which could be
different.

The implementation now ensures, that the header fields are
parsed in the order they have been added.
2015-05-27 17:46:22 +02:00
markharwood 57653df713 Search fix: fielddata_fields query string parameter was ignored.
The RestSearchAction did not parse the fielddata_fields parameter. Added test case and missing parser code.

Closes #11025
2015-05-27 15:30:10 +01:00
Alexander Reelsen fc224a0de8 Cat API: Add wildcard support for header names
This adds wildcard support (simple regexes) for specifying header names.
Aliases are supported as well.

Closes #10811
2015-05-27 16:09:31 +02:00
javanna 44fe99a3a8 [TEST] make filter_path a default parameter in java rest runner
Closes #11351
2015-05-26 15:34:45 +02:00
Tanguy Leroux ce63590bd6 API: Add response filtering with filter_path parameter
This change adds a new "filter_path" parameter that can be used to filter and reduce the responses returned by the REST API of elasticsearch.

For example, returning only the shards that failed to be optimized:
```
curl -XPOST 'localhost:9200/beer/_optimize?filter_path=_shards.failed'
{"_shards":{"failed":0}}%
```

It supports multiple filters (separated by a comma):
```
curl -XGET 'localhost:9200/_mapping?pretty&filter_path=*.mappings.*.properties.name,*.mappings.*.properties.title'
```

It also supports the YAML response format. Here it returns only the `_id` field of a newly indexed document:
```
curl -XPOST 'localhost:9200/library/book?filter_path=_id' -d '---hello:\n  world: 1\n'
---
_id: "AU0j64-b-stVfkvus5-A"
```

It also supports wildcards. Here it returns only the host name of every nodes in the cluster:
```
curl -XGET 'http://localhost:9200/_nodes/stats?filter_path=nodes.*.host*'
{"nodes":{"lvJHed8uQQu4brS-SXKsNA":{"host":"portable"}}}
```

And "**" can be used to include sub fields without knowing the exact path. Here it returns only the Lucene version of every segment:
```
curl 'http://localhost:9200/_segments?pretty&filter_path=indices.**.version'
{
  "indices" : {
    "beer" : {
      "shards" : {
        "0" : [ {
          "segments" : {
            "_0" : {
              "version" : "5.2.0"
            },
            "_1" : {
              "version" : "5.2.0"
            }
          }
        } ]
      }
    }
  }
}
```

Note that elasticsearch sometimes returns directly the raw value of a field, like the _source field. If you want to filter _source fields, you should consider combining the already existing _source parameter (see Get API for more details) with the filter_path parameter like this:

```
curl -XGET 'localhost:9200/_search?pretty&filter_path=hits.hits._source&_source=title'
{
  "hits" : {
    "hits" : [ {
      "_source":{"title":"Book #2"}
    }, {
      "_source":{"title":"Book #1"}
    }, {
      "_source":{"title":"Book #3"}
    } ]
  }
}
```
2015-05-26 13:51:04 +02:00
Boaz Leskes b376a3fbfb Move index sealing terminology to synced flush
#10032 introduced the notion of sealing an index by marking it with a special read only marker, allowing for a couple of optimization to happen. The most important one was to speed up recoveries of shards where we know nothing has changed since they were online by skipping the file based sync phase. During the implementation we came up with a light notion which achieves the same recovery benefits but without the read only aspects which we dubbed synced flush. The fact that it was light weight and didn't put the index in read only mode, allowed us to do it automatically in the background which has great advantage. However we also felt the need to allow users to manually trigger this operation.

 The implementation at #11179 added the sync flush internal logic and the manual (rest) rest API. The name of the API was modeled after the sealing terminology which may end up being confusing. This commit changes the API name to match the internal synced flush naming, namely `{index}/_flush/synced'.

  On top of that it contains a couple other changes:
   - Remove all java client API. This feature is not supposed to be called programtically by applications but rather by admins.
   - Improve rest responses making structure similar to other (flush) API
   - Change IndexShard#getOperationsCount to exclude the internal +1 on open shard . it's confusing to get 1 while there are actually no ongoing operations
   - Some minor other clean ups
2015-05-25 22:32:32 +03:00
Michael McCandless 5aaebb6d83 add missing units to tests 2015-05-25 06:48:34 -04:00
Clinton Gormley 409e4e5f73 REST test: Fixed index-seal test
Fixed bad YAML, and changed to wait for yellow instead of green, because
REST tests usually run on a single node
2015-05-20 13:46:43 +02:00
Simon Willnauer 9d2852f0ab Merge branch 'master' into feature/synced_flush
Conflicts:
	src/main/java/org/elasticsearch/index/engine/InternalEngine.java
	src/main/java/org/elasticsearch/index/shard/IndexShard.java
	src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java
	src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
2015-05-19 12:16:22 +02:00
Adrien Grand 2c241e8a36 Mappings: Remove the `ignore_conflicts` option.
Mappings conflicts should not be ignored. If I read the history correctly, this
option was added when a mapping update to an existing field was considered a
conflict, even if the new mapping was exactly the same. Now that mapping updates
are smart enough to detect conflicting options, we don't need an option to
ignore conflicts.
2015-05-18 15:28:23 +02:00
Martijn van Groningen 719252a138 Merge pull request #11183 from martijnvg/parent-child/remove_id_cache_from_stats_and_clear_cache_apis
Removed `id_cache` from stats and cat apis.
2015-05-15 14:39:35 +02:00
Martijn van Groningen ece18f162e Removed `id_cache` from stats and cat apis.
Also removed the `id_cache` option from the clear cache api.

Closes #5269
2015-05-15 14:06:18 +02:00
Jun Ohtani 3a1a4d3e89 Analysis: Add multi-valued text support
Add support array text as a multi-valued for AnalyzeRequestBuilder
Add support array text as a multi-valued for Analyze REST API
Add docs

Closes #3023
2015-05-15 20:01:10 +09:00
Britta Weber c628d67f9e api for synced flush
closes #11098
2015-05-13 14:45:24 +02:00
javanna d7e585ca4f REST: Unify query_string parameters parsing
There currently are small differences between search api and count, exists, validate query, explain api when it comes to reading query_string parameters.  `analyze_wildcard`, `lowercase_expanded_terms` and `lenient` are only read by the search api and ignored by all other mentioned apis. Unified code to fix this and make sure it doesn't happen again. Also shared some code when it comes to printing out the query as part of SearchSourceBuilder conversion to ToXContent.

Extended REST spec to include all the supported params (some that were already supported weren't listed), and added REST tests (also some basic tests for count and search_exists which weren't tested at all).

Closes #11057
2015-05-11 11:32:59 +02:00
Andrew Selden c953e99324 Merge pull request #10864 from aleph-zero/issues/9606
Remove (dfs_)query_and_fetch from the REST API
2015-05-07 12:51:28 -07:00
Alex Ksikes ec4f12f9ef More Like This: removal of the MLT API
Removes the More Like This API, users should now use the More Like This query.
The MLT API tests were converted to their query equivalent. Also some clean
ups in MLT tests.

Closes #10736
Closes #11003
2015-05-06 18:11:11 +02:00
Adrien Grand b72f27a410 Core: Cut over to the Lucene filter cache.
This removes Elasticsearch's filter cache and uses Lucene's instead. It has some
implications:
 - custom cache keys (`_cache_key`) are unsupported
 - decisions are made internally and can't be overridden by users ('_cache`)
 - not only filters can be cached but also all queries that do not need scores
 - parent/child queries can now be cached, however cached entries are only
   valid for the current top-level reader so in practice it will likely only
   be used on read-only indices
 - the cache deduplicates filters, which plays nicer with large keys (eg. `terms`)
 - better stats: we already had ram usage and evictions, but now also hit count,
   miss count, lookup count, number of cached doc id sets and current number of
   doc id sets in the cache
 - dynamically changing the filter cache size is not supported anymore

Internally, an important change is that it removes the NoCacheFilter infrastructure
in favour of making Query.rewrite specializing the query for the current reader so
that it will only be cached on this reader (look for IndexCacheableQuery).

Note that consuming filters with the query API (createWeight/scorer) instead of
the filter API (getDocIdSet) is important for parent/child queries because
otherwise a QueryWrapperFilter(ParentQuery) would run the wrapped query per
segment while relations might be cross segments.
2015-05-04 09:02:15 +02:00
Honza Král 28e5a649b4 [API] remove global parameter source from individual APIs
Same way we don't define pretty anywhere, we shouldn't have source
2015-04-29 14:01:47 +02:00
aleph-zero 89542facb3 Remove (dfs_)query_and_fetch from the REST API
Remove the ability to specify search type ‘query_and_fetch’ and
‘df_query_and_fetch’ from the REST API.

- Adds REST tests
- Updates REST API spec to remove ‘query_and_fetch’ and
‘df_query_and_fetch’ as options
- Removes documentation for these options

Closes #9606
2015-04-28 15:27:59 -07:00
Michael McCandless cf2fb4ed0f Remove core delete-by-query implementation, to be replaced with a plugin
The current implementation is dangerous: it unexpectedly refreshes,
which can quickly cause an unhealthy index (segment explosion).  It
can also delete different documents on primary vs replicas, causing
inconsistent replicas.

For 2.0 we will replace this with an optional plugin that does a
scan/scroll search and then issues bulk delete requests.

Closes #10859
2015-04-28 16:09:04 -04:00
Simon Willnauer d164526d27 Remove `_shutdown` API
Thsi commit removes the `_shutdown` API entirely without any replacement.
Nodes should be managed from the operating system not via REST APIs
2015-04-27 17:19:36 +02:00
Simon Willnauer 15d58d91f1 [REST] Render REST errors in a structural way
This commit adds support for structural errors / failures / exceptions
on the elasticsearch REST layer. Exceptions are rendering with at least
a `type` and a `reason` corresponding to the exception name and the message.
Some expcetions like the ones associated with an index or a shard will have
additional information about the index the exception was triggered on or the
shard respectivly.

Each rendered response will also contain a list of root causes which is a list
of distinct shard level errors returned for the request. Root causes are the lowest
level elasticsearch exception found per shard response and are intended to be displayed
to the user to indicate the soruce of the exception.

Shard level response are by-default grouped by their type and reason to reduce the amount
of duplicates retunred. Yet, the same exception retunred from different indices will not be
grouped.

Closes #3303
2015-04-24 09:36:10 +02:00
tlrx adc0807c68 Internal: Add METADATA_READ and METADATA_WRITE blocks
This commit splits the current ClusterBlockLevel.METADATA into two disctins ClusterBlockLevel.METADATA_READ and ClusterBlockLevel.METADATA_WRITE blocks. It allows to make a distinction between
an operation that modifies the index or cluster metadata and an operation that does not change any metadata.

Before this commit, many operations where blocked when the cluster was read-only: Cluster Stats, Get Mappings, Get Snapshot, Get Index Settings, etc. Now those operations are allowed even when
the cluster or the index is read-only.

Related to #8102, #2833

Closes #3703
Closes #5855
Closes #10521
Closes #10522
2015-04-23 15:13:57 +02:00
Martijn van Groningen 5705537ecf Added field stats api
The field stats api returns field level statistics such as lowest, highest values and number of documents that have at least one value for a field.

An api like this can be useful to explore a data set you don't know much about. For example you can figure at with the lowest and highest response times are, so that you can create a histogram or range aggregation with sane settings.

This api doesn't run a search to figure this statistics out, but rather use the Lucene index look these statics up (using Terms class in Lucene). So finding out these stats for fields is cheap and quick.

The min/max values are based on the type of the field. So for a numeric field min/max are numbers and date field the min/max date and other fields the min/max are term based.

Closes #10523
2015-04-23 08:52:34 +02:00
Ryan Ernst 2d54738339 Tests: Add shortcut "all" to skip version ranges in rest tests
This was suggested on #10656 as cleaner than " - " to indicate all
versions should be skipped.

closes #10702
2015-04-22 11:40:26 -07:00
Boaz Leskes 1118965aa5 Test: move to ensure yellow in indices.stats/12_level.yaml 2015-04-22 16:10:28 +02:00
Boaz Leskes 91ff3f6963 Test: add ensure green to indices.stats/12_level.yaml
Also changed the stash logger to not log all stashed values under debug (it does trace now) but do dump the stash content upon failure (under info as a XContent)
2015-04-22 15:56:45 +02:00
Jun Ohtani 0955c127c0 Rest: Add json in request body to scroll, clear scroll, and analyze API
Change analyze.asciidoc and scroll.asciidoc
Add json support to Analyze and Scroll, and clear scrollAPI
Add rest-api-spec/test

Closes #5866
2015-04-22 17:53:20 +09:00
Boaz Leskes a1ba339517 Stats: add CommitStats to supply information about the current commit point
Extends ShardStats with commit specific information. We currently expose commit id, generation and the user data map.

The information is also retrievable via the Rest API by using `GET _stats?level=shards`

Closes #10687
2015-04-22 09:33:14 +02:00
Adrien Grand d7abb12100 Replace deprecated filters with equivalent queries.
In Lucene 5.1 lots of filters got deprecated in favour of equivalent queries.
Additionally, random-access to filters is now replaced with approximations on
scorers. This commit
 - replaces the deprecated NumericRangeFilter, PrefixFilter, TermFilter and
   TermsFilter with NumericRangeQuery, PrefixQuery, TermQuery and TermsQuery,
   wrapped in a QueryWrapperFilter
 - replaces XBooleanFilter, AndFilter and OrFilter with a BooleanQuery in a
   QueryWrapperFilter
 - removes DocIdSets.isBroken: the new two-phase iteration API will now help
   execute slow filters efficiently
 - replaces FilterCachingPolicy with QueryCachingPolicy

Close #8960
2015-04-21 15:32:43 +02:00
Adrien Grand 1adf232bb2 Mappings: Validate dynamic mappings updates on the master node.
This commit changes dynamic mappings updates so that they are synchronous on the
entire cluster and their validity is checked by the master node. There are some
important consequences of this commit:
 - a failing index request on a non-existing type does not implicitely create
   the type anymore
 - dynamic mappings updates cannot create inconsistent mappings on different
   shards
 - indexing requests that introduce new fields might induce latency spikes
   because of the overhead to update the mappings on the master node

Close #8688
2015-04-21 11:08:21 +02:00
Ryan Ernst 68f75ea7b6 simplified rest skip range version parsing more, ranges can now be open
ended
2015-04-19 06:50:08 -07:00
Ryan Ernst 06eee11dbb simplify version handling in rest tests, add tests for version utilities 2015-04-18 12:48:00 -07:00
Michael McCandless 399f0ccce9 Core: add only_ancient_segments to upgrade API, so only segments with an old Lucene version are upgraded
This option defaults to false, because it is also important to upgrade
the "merely old" segments since many Lucene improvements happen within
minor releases.

But you can pass true to do the minimal work necessary to upgrade to
the next major Elasticsearch release.

The HTTP GET upgrade request now also breaks out how many bytes of
ancient segments need upgrading.

Closes #10213

Closes #10540

Conflicts:
	dev-tools/create_bwc_index.py
	rest-api-spec/api/indices.upgrade.json
	src/main/java/org/elasticsearch/action/admin/indices/optimize/OptimizeRequest.java
	src/main/java/org/elasticsearch/action/admin/indices/optimize/ShardOptimizeRequest.java
	src/main/java/org/elasticsearch/action/admin/indices/optimize/TransportOptimizeAction.java
	src/main/java/org/elasticsearch/index/engine/InternalEngine.java
	src/test/java/org/elasticsearch/bwcompat/StaticIndexBackwardCompatibilityTest.java
	src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
	src/test/java/org/elasticsearch/rest/action/admin/indices/upgrade/UpgradeReallyOldIndexTest.java
2015-04-16 05:24:33 -04:00
Simon Willnauer b756477fb4 [CORE] Rename START phase into VERIFY_INDEX
we really only optionally run checkindex in this phase and moving
the engine start into translog is move cleaner code wise.

Closes #10570
2015-04-14 09:53:02 +02:00
Spencer Alger 0cd1848e88 Indexed scripts/templates: fix yaml test indentation 2015-04-08 15:36:23 -07:00
javanna acabf2d55a Cluster state REST api: print routing_nodes out only when requested through specific flag
For bacwards compatibility reasons routing_nodes were previously printed out when routing_table was requested, together with the actual routing_table. Now they are printed out only when requests through `routing_nodes` flag.

Relates to #10412
Closes #10486
2015-04-08 16:10:36 +02:00
Leonardo Menezes 5fd9aee16e Cluster state REST api: routing_nodes as an independent metric option
Cluster state api returns both routing_table and routing_nodes sections whenever routing_table is requested. That is pretty much the same info, just grouped differently. This commit allows to differentiate between the two. Yet, routing_table still returns both for bw comp reasons.

Closes #10352
Closes #10412
2015-04-08 15:35:11 +02:00
javanna df875707ec Indexed scripts/templates: return response body when script is not found
Align get indexed scripts and get search template apis to our get api, which returns a response body when the document is not found, with a found boolean flag. Also, return metadata info all the time too.

Closes #7325
Closes #10396
2015-04-08 10:44:11 +02:00
Christoph Büscher 2beda3953d Remove expansion of empty index arguments in RoutingTable
RoutingTables activePrimaryShardsGrouped(), allActiveShardsGrouped() and
allAssignedShardsGrouped() methods treated empty index array input
parameters as meaning "all" indices and expanded to the routing maps
keyset. However, the expansion of index names is now already done in
MetaData#concreteIndices(). Returning an empty index name list here
when a wildcard pattern didn't match any index name could lead to
problems like #9081 because the RoutingTable still expanded this
list of names to "_all". In case of e.g. the recovery endpoint this
could lead to problems.

Closes #9081
Closes #10148
2015-04-02 12:29:47 +02:00
Adrien Grand a608db122d Search: Remove the `count` search type.
This commit brings the benefits of the `count` search type to search requests
that have a `size` of 0:
 - a single round-trip to shards (no fetch phase)
 - ability to use the query cache

Since `count` now provides no benefits over `query_then_fetch`, it has been
deprecated.

Close #7630
2015-03-31 11:31:49 +02:00
Martijn van Groningen 6d1a1b328b Make sure that the parent option on the update request only is delgated to upsert index request.
Closes #4538
2015-03-28 08:53:11 +01:00
Martijn van Groningen 75713f4190 Reverted commit: 20f7be3 2015-03-28 08:53:11 +01:00
Leonardo Menezes 5feaa484b5 Cat api: added node id column to _cat/plugins response
relates to #10226
2015-03-25 12:00:38 +01:00
Leonardo Menezes d086648cec Cat api: added node id column to _cat/segments response
Relates to #10226
2015-03-25 12:00:15 +01:00
Leonardo Menezes bad60cea13 Cat api: added node id column to _cat/shards response
closes #10226
2015-03-25 11:59:35 +01:00
Clinton Gormley 1ee42e93d0 Fix REST test delete/42_missing_parent to catch a RoutingMissingException
Related to #10136
2015-03-25 06:48:05 +02:00
Ryan Ernst 90dfd78267 Remove missed references to delete mapping API
See #10231
2015-03-24 10:13:19 -07:00
Ryan Ernst 693d91e41c Mappings: Remove delete mapping API
Deleting a type from an index is inherently dangerous because
the type can be recreated with new mappings which may conflict
with existing segments still using the old mappings. This
removes the ability to delete a type (similar to how deleting
fields within a type is not allowed, for the same reason).

closes #8877
closes #10231
2015-03-24 09:46:02 -07:00
Honza Král 9c0f655674 [API] changed all the links in json specs to the new site 2015-03-24 17:30:32 +01:00
Tim Schlechter 56117e3f70 [SPEC] Remove duplicated timeout param from bulk REST spec
Closes #10205
2015-03-22 10:26:12 +01:00
javanna 481ef7d3f4 [TEST] add REST test for update index settings api using indices options
Relates to #10030
2015-03-21 11:10:24 +01:00
Simon Willnauer 7257345db9 Revert Benchmark API
The benchmark api is being worked on feature/bench branch and will be merged from there when ready.
2015-03-21 10:36:04 +01:00
Boaz Leskes 8c69535580 Recovery: add throttle stats
This commit adds throttling statistics to the index stats API and to the recovery state.

Closes #10097
2015-03-20 10:13:40 +01:00
Simon Willnauer 1168347b9d [REPLICATION] Remove `async` replication
Closes #10114
2015-03-19 14:44:21 -07:00
Clinton Gormley 6e9eea9b98 Deleted REST test pending shard header
Bug introduced in #10162
2015-03-19 17:57:25 +01:00
Clinton Gormley 25369f0727 Remove async replication from the docs and REST spec
Relates to #10114
2015-03-19 15:34:12 +01:00
Christoph Büscher 5788289a6d [TEST] Remove 'gtelte' feature from yaml tests
Some yaml tests still used the 'gtelte' feature in their skip section.
Since all language clients support this, the feature skip can be
removed.
2015-03-18 15:31:17 -07:00
Boaz Leskes b605184471 Recovery: add total operations to the `_recovery` API
This commit adds the current total number of translog operations to the recovery reporting API. We also expose the recovered / total percentage:

```
"translog": {
   "recovered": 536,
   "total": 986,
   "percent": "54.3%",
   "total_time": "2ms",
   "total_time_in_millis": 2
},
```

Closes #9368
Closes #10042
2015-03-17 07:31:29 -07:00
Lee Hinman 6aec68cd29 Revert "[QUERY] Remove lowercase_expanded_terms and locale options"
This reverts commit d1f7bd97cb.

Ryan pointed out that this needs to work with the multi term query, so
additional analysis and tests should be added.
2015-03-13 13:51:44 -06:00
Lee Hinman d1f7bd97cb [QUERY] Remove lowercase_expanded_terms and locale options
The analysis chain should be used instead of relying on this, as it is
confusing when dealing with different per-field analysers.

The `locale` option was only used for `lowercase_expanded_terms`, which,
once removed, is no longer needed, so it was removed as well.

Fixes #9978
Relates to #9973
2015-03-13 13:17:27 -06:00
Lee Hinman 583c4926c4 Mark shadow replicas with 's' in _cat/shards output
Fixes #9772
2015-03-06 13:21:29 -07:00
javanna 4ad33c30ac [TEST] Work around URI encode limitations in RestClient
We've been relying on URI for url encoding, but it turns out it has some problems. For instance '+' stays as is while it should be encoded to `%2B`. If we go and manually encode query params we have to be careful though not to run into double encoding ('+'=>'%2B'=>'%252B'). The applied solution relies on URI encoding for the url path, but manual url encoding for the query parameters. We prevent URI from double encoding query params by using its single argument constructor that leaves everything as is.

We can also revert back the expression script REST test that revealed this to its original content (which contains an addition).

Closes #9769
Closes #9946
2015-03-03 10:21:23 +01:00
Clinton Gormley 8cba6509e4 REST: Fix script-expressions test to work with Perl
Closes #9931
2015-03-02 10:27:05 +01:00
Clinton Gormley 03d6dae834 REST tests - disable expressions test for now
Java returns floating point but JSON contains integer
2015-02-28 19:31:28 +01:00
Clinton Gormley 53d0a0a4b9 REST test: Fixed expressions test
Perl reads the number as a string, so instead mapped the field as a long
specifically.

Also, the value returned in JSON is an integer, not a float
2015-02-28 17:02:29 +01:00
Clinton Gormley 4d78026ef1 Updated copyright years to include 2015 2015-02-28 03:18:45 +01:00
Ryan Ernst dd0084de83 Mappings: Lock down _ttl field
While the parser allowed changing field type settings, these would never
have been serialized.  So this change simply removes parsing using
parseField. Backcompat will still work if a user uploads old settings
(they just would never have worked anyways, so we continue ignoring
them with 1.x, and 2.x will now error).

see #8143
closes #9914
2015-02-27 11:11:04 -08:00
Boaz Leskes cc2b00c443 Test: cluster.health rest test should use 0 replicas
Some client test harnesses do not limit the number of replicas based on the number of nodes.
2015-02-25 20:35:39 +01:00
Boaz Leskes 3e32dd985a Recovery: RecoveryState clean up
To support the `_recovery` API, the recovery process keeps track of current progress in a class called RecoveryState. This class currently have some issues, mostly around concurrency (see #6644 ). This PR cleans it up as well as other issues around it:

- Make the Index subsection API cleaner:
- remove redundant information - all calculation is done based on the underlying file map
- clearer definition of what is what: total files, vs reused files (local files that match the source) vs recovered files (copied over). % based progress is reported based on recovered files only.
- cleaned up json response to match other API (sadly this breaks the structure). We now properly report human values for dates and other units.
- Add more robust unit testing
- Detail flag was passed along as state (it's now a ToXContent param)
- State lookup during reporting is now always done via the IndexShard , no more fall backs to many other classes.
- Cleanup APIs around time and move the little computations to the state class as opposed to doing them out of the API

I also improved error messages out of the REST testing infra for things I run into.

Closes #6644
Closes #9811
2015-02-25 17:34:22 +01:00
Boaz Leskes 06417a85e0 Test: cat health api didn't disable headers 2015-02-25 16:17:47 +01:00
Boaz Leskes 6953777c3a API: add pending tasks count to cluster health
The number of current pending tasks is useful to detect and overloaded master. This commit adds it to the cluster health API. The complete list can be retrieved from the dedicated pending tasks API.

It also adds rest tests for the cluster health variants.

Closes #9877
2015-02-25 14:58:44 +01:00
Alexander Reelsen 5de564a19f Testing: Remove plus sign from YAML test due to encoding issue
The plus sign is not treated correctly in encoding and can lead
to problems, if the search request is encoded as HTTP parameter
instead of the HTTP body.

Relates #9769
2015-02-19 18:35:39 +01:00
Alexander Reelsen 2dc70597c2 Testing: Added REST tests to check if expression scripting works 2015-02-19 09:15:28 +01:00
Boaz Leskes c9480783b3 Rest: expose `master_timeout` flag on `GET _template` & `HEAD _template`
These are master level operation but we currently don't expose this parameter

Closes #9688
2015-02-13 11:25:26 +01:00
Clinton Gormley 75e7ce9c51 REST Test: Lucene segment format can have two or three numbers: 4.10.2 or 4.9 2015-02-12 19:59:43 +01:00
Clinton Gormley 93fe4495ae REST tests - add feature groovy_scripting to skip tests if groovy not enabled 2015-02-12 18:08:35 +01:00
Clinton Gormley 36758ce81b Fixed cat.segments and indices.segments tests - bad YAML indenting 2015-02-12 17:35:19 +01:00
Christoph Büscher 41befaf6b5 Change behaviour of indices segments api to allow no indices
Using '_cat/segments' or the indices segments api without matching any index
now returns empty result instead of throwing IndexMissingException.

Closes #9219
2015-02-12 14:42:15 +01:00
javanna 4e94be8a37 [TEST] Introduce basic validation of our REST spec
Whenever we have an api that supports GET with a body, we always support the POST method too, as well as providing the body as a query_string parameter called `source`. Our REST spec should reflect this convention. FIxed them and introduced a hard check at parse time in our Java REST tests runner, which will cause the tests to fail if spec are not compliant.

Closes #9629
2015-02-12 22:25:17 +11:00
javanna 9c847db8af Percolate api: support encoded body as query string param consistently
The percolate api doesn't parse the encoded body provided as `source` query string parameter, when percolating an existing document. Fixed and added REST test that would have caught this since we randomly use GET + encoded `source` param instead of GET + request body in our java runner (the perl runner does the same too).

Closes #9628
2015-02-11 08:53:04 +11:00
Ryan Ernst c6968883a7 Mappings: Remove support for new indexes using path setting in
object/nested fields or index_name in any field

Backcompat is still here for indexes created before 2.0.

closes #6677
2015-02-05 12:44:43 -08:00
Simon Willnauer 0c5599e1d1 [ENGINE] Remove full flush / FlushType.NEW_WRITER
The `full` option and `FlushType.NEW_WRITER` only exists to allow
realtime changes to two settings (`index.codec` and `index.concurrency`).
Those settings are very expert and don't really need to be updateable
in realtime.
2015-02-04 17:38:05 +01:00
Clinton Gormley 988f35a7da REST: Add skip "stash_in_path" feature to nodes.info/20_transport test
Required for clients that don't yet do stash lookups when resolving
fieldnames like "nodes.$master.transport"
2015-02-02 17:11:04 +01:00
Clinton Gormley e7ac5f296e REST-spec: Allow stashed values to be used in property names as well
Fix the nodes.info/20_transport test to use the master node, rather
than to rely on applying a regex to the whole $body
2015-02-02 16:12:41 +01:00
Alexander Reelsen 59f8c0951a Netty Transport: Add profiles to transport infos
Until now, there was no possibility to expose infos about configured
transport profiles. This commit adds the ability to expose those
information in the TransportInfo class.

The channel was well as the netty pipeline handler now also contain
the profile they were configured for, as this information cannot be
extracted elsewhere.

In addition, each profile now can set its own publish host and port,
which might be needed in case of portforwarding or using docker.

Closes #9134
2015-02-02 08:17:55 +01:00
Andrew Ochsner f95fa83e5b Closes #9398 2015-01-23 18:06:12 +01:00
Clinton Gormley 6aff2b9f65 Changed all rest-spec definitions of expand_wildcards to be an enum
with values: open, closed, none, all

Relates to #9295
2015-01-23 16:05:59 +01:00
Christoph Büscher eeb96db76b Rest: Adding support of multi-index query parameters for _cluster/state
Adding missing support for the multi-index query parameters 'ignore_unavailable',
'allow_no_indices' and 'expand_wildcards' to '_cluster/state' API. These
parameters are supposed to be supported for APIs that work across multiple indices.
So far overwriting the default settings per REST call was not possible which is
fixed here.

Closes #5229
Closes #9295
2015-01-22 16:47:14 +01:00
David Pilato 052645903a Rest: remove status code from main action
Today we give the HTTP status back within the HTTP response itself and within the JSON response as well:

```sh
curl localhost:9200/
```

```js
{
  "status" : 200,
  "name" : "Red Wolf",
  "version" : {
    "number" : "2.0.0",
    "build_hash" : "6837a61d8a646a2ac7dc8da1ab3c4ab85d60882d",
    "build_timestamp" : "2014-08-19T13:55:56Z",
    "build_snapshot" : true,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}
```
2015-01-12 12:37:46 +01:00
Martijn van Groningen a88dd36df4 Test: Only run the test that verifies the `pending` field in the `_shards` header if the test cluster has two are more data nodes. 2015-01-12 10:29:23 +01:00
Christoph Büscher 04cb09f44c [TEST] Add missing docs and tests for '_cat/segments'
The '_cat/segments' api was missing docs and a rest test which are added here.

Closes #5856
2015-01-09 12:29:11 +01:00
Martijn van Groningen ca4f27f40e Core: Added `_shards` header to all write responses.
The header indicates to how many shard copies (primary and replicas shards) a write was supposed to go to, to how many
shard copies to write succeeded and potentially captures shard failures if writing into a replica shard fails.

For async writes it also includes the number of shards a write is still pending.

Closes #7994
2015-01-08 18:10:08 +01:00
Colin Goodheart-Smithe ecfe72ebcc Indices API: Fix to make GET Index API consistent with docs
This fix ensures that calls to the GET alias/mappings/settings/warmers APIs return the aliases/mappings/settings/warmers object even if there is no content within them.. This make them consistent with the GET Index API docs and the breaking changes in 1.4 docs

Closes #9148
2015-01-08 08:48:44 +00:00
Martijn van Groningen 20f7be378b Removed parent parameter from update request, because it is just sets the routing.
The routing option should be used instead. The parent a child document points to can't be updated.

Closes #4538
2015-01-07 10:26:20 +01:00
Michael McCandless 242e631e95 Core: ignore known idle threads by default in /_nodes/hot_threads
Add a new ignore_idle_threads boolean option (default true) to
/_nodes/hot_threads, to filter out threads in known idle places like
waiting on a socket select or on pulling the next task from an empty
queue.

Closes #8985

Closes #8908
2014-12-17 11:59:31 -05:00
Alex Ksikes 86e1655e4b Term Vectors: support for version and version_type
This commit adds support for version and version_type to the Term Vectors API.
This could be useful in the following case whereby the user gets a document
and later wants to generate its TVs. With version, this would ensure that only
the TVs of that particular document are generated, and error out if the
document has been updated in between.

Closes #7480
2014-12-17 15:43:15 +01:00
Martijn Laarman bc76032fdd Documented the new terminate_after querystring option on search as implemented in #6885 2014-12-17 14:49:05 +01:00
Lee Hinman 9b18c44b67 Default _cat APIs to verbose
`?v=false` can be used if the headers are not desired.

Resolves #8922

Fixes #8927
2014-12-15 12:51:59 +00:00
Martijn Laarman bb6e539bc7 updated (m)termvectors api specs to reflect #7530 which made it into 1.4 2014-12-12 11:04:21 +01:00
Honza Král 750310cf94 [API] indices.upgrade's wait_for_completion param defaults to false 2014-12-11 17:29:14 +01:00
Sebastian Ziebell ab001cf8e3 API endpoint cluster.put_settings accepts timeout & master_timeout query parameters
Closes #8769
2014-12-04 09:14:20 +01:00
Alex Ksikes 256712640f MLT Query: Support for ignore docs
Adds a `ignore_like` parameter to the MLT Query, which simply tells the
algorithm to skip all the terms from the given documents. This could be useful
in order to better guide nearest neighbor search by telling the algorithm to
never explore the space spanned by the given `ignore_like` docs. In essence we
are interested about the characteristic of a given item, but not of the ones
provided by `ignore_like`, thereby forcing the algorithm to go deeper in its
selection of terms. Note that this is different than simply performing a must
not boolean query on the unliked items. The syntax is exactly the same as the
`like` parameter.

Closes #8674
2014-11-28 14:48:43 +01:00
Sebastian Ziebell 4dd228d02f Spec: Fixes URL links to documentation
Closes #8656
2014-11-25 19:23:59 +01:00
Karel Minarik d33f1d4bd8 [SPEC] Removed the utilities for generating REST specs/code
The utilities have been migrated to the elasticsearch/elasticsearch-ruby repository.

See: https://github.com/elasticsearch/elasticsearch-ruby/tree/master/elasticsearch-api/utils

Related: elasticsearch/elasticsearch-ruby@dfac51d

Closes #8606
2014-11-25 15:33:44 +01:00
Michael McCandless 856b294441 Core: let Lucene kick off merges
Today, Elasticsearch has a separate merge thread pool checking once
per second (by default) if any merges are necessary, but this is no
longer necessary since we can and do now tell Lucene's
ConcurrentMergeScheduler never to "hard pause" threads when merges
fall behind, since we do our own index throttling.

This change goes back to letting Lucene launch merges as needed, and
removes these two expert settings:

  index.merge.force_async_merge
  index.merge.async_interval

Now merges kick off immediately instead of waiting up to 1 second
before running.

Closes #8643
2014-11-25 04:13:57 -05:00
Sebastian Ziebell e0d6e0a6d3 Spec: Fixes a few locations of the params array in JSON specification
A few locations of the `params` object in the JSON specification were not located in the `url` block.

Closes #8607
2014-11-24 20:43:33 +01:00
Alex Ksikes 1959275622 Term Vectors: More consistent naming for term vector[s]
We speak of the term vectors of a document, where each field has an associated
stored term vector. Since by default we are requesting all the term vectors of
a document, the HTTP request endpoint should rather be called `_termvectors`
instead of `_termvector`. The usage of `_termvector` is now deprecated, as
well as the transport client call to termVector and prepareTermVector.

Closes #8484
2014-11-21 14:06:44 +01:00
nwarz e77f9720d2 Mapping: Fixes Merging of default analyzer
Fixed behaviour where two representations of the default index analyzer weren't being treated as equivalent. Added REST test to confirm fix.

Closes #2716
2014-11-14 09:58:26 +00:00
Clinton Gormley f9810e591e Rest API: indices.delete should have "index" as a required param 2014-11-10 17:06:43 +01:00
Chris Mattmann f47fb6b1cf fix mention of Apache License, version 2.
Closes #8400
2014-11-08 19:14:51 +01:00
Veres Lajos 4059e4ac86 typo fixes - https://github.com/vlajos/misspell_fixer
Closes #8323
2014-11-08 18:55:57 +01:00
javanna 3afe86d447 [TEST] fix term_vector realtime test to call wait for green and use no replicas
If a shard (e.g. replica) gets initialized after we indexed the document it gets refreshed internally and we find the doc and its term_vectors, thus the test fails
2014-11-07 20:05:14 +01:00
Adrien Grand 9ea25df649 Switch to murmurhash3 to route documents to shards.
We currently use the djb2 hash function in order to compute the shard a
document should go to. Unfortunately this hash function is not very
sophisticated and you can sometimes hit adversarial cases, such as numeric ids
on 33 shards.

Murmur3 generates hashes with a better distribution, which should avoid the
adversarial cases.

Here are some examples of how 100000 incremental ids are distributed to shards
using either djb2 or murmur3.

5 shards:
Murmur3: [19933, 19964, 19940, 20030, 20133]
DJB:     [20000, 20000, 20000, 20000, 20000]

3 shards:
Murmur3: [33185, 33347, 33468]
DJB:     [30100, 30000, 39900]

33 shards:
Murmur3: [2999, 3096, 2930, 2986, 3070, 3093, 3023, 3052, 3112, 2940, 3036, 2985, 3031, 3048, 3127, 2961, 2901, 3105, 3041, 3130, 3013, 3035, 3031, 3019, 3008, 3022, 3111, 3086, 3016, 2996, 3075, 2945, 2977]
DJB:     [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 900, 900, 900, 900, 1000, 1000, 10000, 10000, 10000, 10000, 9100, 9100, 9100, 9100, 9000, 9000, 0, 0, 0, 0, 0, 0]

Even if djb2 looks ideal in some cases (5 shards), the fact that the
distribution of its hashes has some patterns can raise issues with some shard
counts (eg. 3, or even worse 33).

Some tests have been modified because they relied on implementation details of
the routing hash function.

Close #7954
2014-11-04 16:32:42 +01:00
Martijn Laarman 006acfe8bf Added missing percolate API parameters (percolate_routing, percolate_preference) to the REST API Spec
Closes #7173
2014-10-28 19:34:34 +01:00
Clinton Gormley b94b3b2bcd Tests: Refixed the put_alias tests
The missing/null index parameter is in the URL, not the body.
So it should be required, and throw a param error if not provided.

Relates #7863
2014-10-28 13:39:05 +01:00
Clinton Gormley f9b5906871 Test: Change missing/null alias test to catch a request error, not a param error
Relates: #7863
2014-10-28 11:43:10 +01:00
Spencer 055e766be7 Updated paths to be inline with #8240 2014-10-27 23:08:19 -07:00
tlrx 8c864cf3f6 Cat Recovery API: Reverting changes introduced with commit e1c75bae87
Adding these 2 headers to the CAT Recovery made the CI tests hanging for a loooong time.

Related to #8041
2014-10-27 20:49:58 +01:00
Zachary Tong f5b2dfd052 Aliases: Throw exception if index is null or missing when creating an alias
Fixes a bug where alias creation would allow `null` for index name, which thereby
applied the alias to _all_ indices.  This patch makes the validator throw an
exception if the index is null.

```bash
POST /_aliases
{
   "actions": [
      {
         "add": {
            "alias": "empty-alias",
            "index": null
         }
      }
   ]
}
```
```json
{
   "error": "ActionRequestValidationException[Validation Failed: 1: Alias action [add]: [index] may not be null;]",
   "status": 400
}
```

The reason this bug wasn't caught by the existing tests is because
the old test for nullness only validated against a cluster which had
zero indices.  The null index is translated into "_all", and since
there are no indices, this fails because the index doesn't exist.
 So the test passes.

However, as soon as you add an index, "_all" resolves and you get the
situation described in the original bug report:  null index is
accepted by the alias, resolves to "_all" and gets applied to everything.

The REST tests, otoh, explicitly tested this bug as a real feature and therefore
passed.  The REST tests were modified to change this behavior.

Fixes #7863
2014-10-27 14:39:01 -04:00
tlrx 96e62b3c1b [TESTS Fix wrong assertion in test introduced by #8128 2014-10-27 11:11:54 +01:00
tlrx e1c75bae87 Cat API: Add node name to _cat/recovery
Add source_node and target_node fields to the recovery cat API. Also fixed and updated the documentation which was not complete concerning fields names.

Closes #8041
2014-10-27 09:47:26 +01:00
Alex Ksikes c13f5f21de Term Vectors: support for distributed frequencies
Adds distributed frequencies support for the Term Vectors API. A new parameter
called `dfs` is introduced which defaults to `false`.

Closes #8144
2014-10-23 13:59:59 +02:00
Adrien Grand f4ee3f25e4 Mappings: Store _timestamp by default.
Storing `_timestamp` by default means that under the default configuration, you
would have all the information you need in order to reindex into a different
index.

Close #8139
2014-10-20 12:17:26 +02:00
Spencer 249a145a5c Update cat.nodes regex
Support for multi-word node names like 'Henry "Hank" McCoy'
2014-10-15 14:33:41 -07:00
Chris Earle 04926954e2 Updating test to support Windows file descriptor count (-1) 2014-10-15 14:02:32 -05:00
Chris Earle 29c5aaa1d3 Fixing test to allow decimal numbers for load. 2014-10-15 11:54:12 -05:00
Chris Earle 2d8a140ed8 Add file descriptor details to cat/nodes
cat/nodes currently does not report any details related to file descriptors. This adds the current number in use, the maximum number available as well as their ratio (percentage) to cat/nodes as hidden-by-default metrics. In addition, this also adds current heap usage (as a non-percentage of ts max) and ram usage (as a non-percerntage of its max) to allow tools to provide more granularity.

Closes #7652
2014-10-15 10:18:41 -05:00
Clinton Gormley dfcc0f97f0 Spec: Removed flush and max_num_segments from indices.upgrade 2014-10-11 17:19:06 +02:00
Clinton Gormley 1e47f02891 Rest: Added missing parameters to indices.upgrade 2014-10-11 16:40:45 +02:00
Clinton Gormley 20a901964c Spec: Added human flag to indices.get_upgrade 2014-10-10 17:26:36 +02:00
Karel Minarik b1d4cec7ab [SPEC] Separated the "Upgrade Index" API into two methods
* `get_upgrade` => `GET _upgrade`  -- Return the status
* `upgrade`     => `POST _upgrade` -- Perform the operation

Original specification part of c021f22523.

Related: #7884, #7922
2014-10-09 16:19:58 +02:00
Ryan Ernst c06c10bbb0 Remove deprecations from master (follow up to #7922) 2014-10-07 08:35:11 -07:00
Ryan Ernst c021f22523 Add Upgrade API
This commit does the following:
* Add the new API at the rest layer, being backed by the optimize API
  with upgrade flag, and segments api to find upgrade status.
* Add `upgrade` flag to optimize API, and deprecate `force` flag (will
  remove in master)
* Add test for both synchronous and async upgrade

closes #7884
closes #7922
2014-10-07 08:09:50 -07:00
Igor Motov 555bfcb02b [SNAPSHOT] Add repository validation
Fixes #7096
2014-10-07 10:50:16 -04:00
David Pilato f0052a58d6 Admin: show open and closed indices in _cat/indices
When asking for `GET /_cat/indices?v`, you can now retrieve closed indices in addition to opened ones.

```
health status index              pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .marvel-2014.05.21   1   1       8792            0     21.7mb         21.7mb
       close  test
yellow open   .marvel-2014.05.22   1   1       3871            0     10.7mb         10.7mb
red    open   .marvel-2014.05.27   1   1
```

Closes #7907.
Closes #7936.
2014-10-03 13:16:14 +02:00
Alex Ksikes c4830cf862 Term Vectors: support for realtime
By default term vectors are now realtime, as opposed to previously near
realtime. If they are not found in the index, they will be generated on the
fly. The document is fetched from the transaction log and treated as an
artificial document. One can set `realtime` parameter to `false` in order to
disable this functionality. This consequently makes the MLT query realtime in
fetching documents, as it previsouly used to be before switching from using
the multi get API to the mtv API.

Closes #7846
2014-10-03 09:26:47 +02:00
Honza Král 3aa4ac9caa [TEST] Make mlt rest tests pass on a single node cluster 2014-09-30 22:42:51 +02:00
Alex Ksikes b118558962 MLT Query: Support for artificial documents
Previously, the only way to specify a document not present in the index was to
use `like_text`. This would usually lead to complex queries made of multiple
MLT queries per document field. This commit adds the ability to the MLT query
to directly specify documents not present in the index (artificial documents).
The syntax is similar to the Percolator API or to the Multi Term Vector API.

Closes #7725
2014-09-29 15:49:13 +02:00
Spencer 19f5a86c21 Update indices.get_mapping.json 2014-09-25 09:53:27 -07:00
javanna 07ca08dbed [TEST] improved regular scroll REST test
Added sort clause by field and checked docs returned each time
2014-09-25 13:02:16 +02:00
javanna dc1ef7e670 [TEST] improved regular scroll REST test
The intermediate document that gets indexed has now a non existing id, and we make sure it doesn't get returned.
2014-09-25 11:46:05 +02:00
javanna f52375198b [TEST] add regular scroll REST test
Closes #7860
2014-09-25 11:03:12 +02:00
Brian Murphy 8e742c2096 Indexed Scripts/Templates : Cleanup
This contains several cleanups to the indexed scripts.
Remove the unused FetchSourceContext from the Get request..
Add lang,_version,_id to the REST GET API.
Removes the routing from GetIndexedScriptRequest since the script index is a single shard that is replicated across all nodes.
Fix backward compatible template file reference
Before 1.3.0 on disk scripts could be referenced by requesting
````
_search/template

{
  "template" : "ondiskscript"
}
````
This was broken in 1.3.0 by requiring
````
{
  "template" :
  {
    "file" : "ondiskscript"
  }
}
````
This commit restores the previous behavior.
Remove support for preference, realtime and refresh
These parameters don't make sense anymore for indexed scripts as we always force the preference to _local and
always refresh after a Put to the indexed scripts index.

Closes #7568
Closes #7559
Closes #7647
Closes #7567
2014-09-19 11:59:08 +01:00
javanna 7e0481d906 More Like This API: remove unused search_query_hint parameter
Closes #7691
2014-09-11 17:34:54 +02:00