Commit Graph

1249 Commits

Author SHA1 Message Date
Lee Hinman 3b5058017e Merge branch 'remove-optimize-rest' 2015-10-29 15:18:03 -06:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
xuzha 97ecd7bf5a Expose pending cluster state queue size in node stats
Add 3 stats about the queue: total queue size, number of committed cluster
states, and number of pending cluster states.
2015-10-28 10:59:15 -07:00
Lee Hinman 3a458af0b7 Remove /_optimize REST API endpoint
The `/_optimize` endpoint was deprecated in 2.1.0 and can now be removed
entirely.
2015-10-27 10:17:16 -06:00
javanna dc900a08a6 Remove "query" query and fix related parsing bugs
We have two types of parse methods for queries: one for the inner query, to be used once the parser is positioned within the query element, and one for the whole query source, including the query element that wraps the actual query.

With the search refactoring we ended up using the former in count, cat count and delete by query, whereas we should have used the former.  It ends up working properly given that we have a registered (deprecated) query called "query", which used to allow to wrap a filter into a query, but this has the following downsides:
1) prevents us from removing the deprecated "query" query
2) we end up supporting a top level query that is not wrapped within a query element (pre 1.0 syntax iirc that shouldn't be supported anymore)

This commit finally removes the "query" query and fixes the related parsing bugs. We also had some tests that were providing queries in the wrong format, those have been fixed too.

Closes #13326
Closes #14304
2015-10-27 14:54:30 +01:00
Yannick Welsch 5959058719 Unique repository names for rest tests 2015-10-26 21:02:25 +01:00
Yannick Welsch 2f10300a03 Merge pull request #14247 from ywelsch/feature/cat-snapshots
Add cat API for repositories and snapshots
2015-10-26 18:40:39 +01:00
Yannick Welsch ca75b7b6ce Add cat API for repositories and snapshots
Closes #14247
Closes #13919
2015-10-26 18:37:10 +01:00
javanna 75cedca0da Remove search exists api
Closes #13682
Closes #13911
2015-10-21 17:39:32 +02:00
Lee Hinman 9ea4909035 Add Force Merge API, deprecate Optimize API
This adds an API for force merging lucene segments. The `/_optimize` API is now
deprecated and replaced by the `/_forcemerge` API, which has all the same flags
and action, just a different name.
2015-10-20 09:00:24 -06:00
Spencer ff9999876c [REST tests] prevent backtracking in cat.nodeattrs 2015-10-19 09:36:19 -07:00
Colin Goodheart-Smithe 4557d1b560 Merge branch 'master' into feature/search-request-refactoring
# Conflicts:
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/FunctionScoreTests.java
2015-10-08 11:39:34 +01:00
Igor Motov a358f34276 Expose nodes operation timeout in REST API
Currently it's not possible to specify a timeout for nodes operations (such as node info, node stats, cluster stats and hot threads) via REST-based APIs.
2015-10-07 18:07:59 -04:00
Robert 331d2d9955 Update update.json
Missing spec for [detect_noop](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html?q=update%20a#_literal_detect_noop_literal) parameter.
2015-10-07 16:13:31 -04:00
Boaz Leskes bcb3fab6ac Engine: Remove Engine.Create
The `_create` API is handy way to specify an index operation should only be done if the document doesn't exist. This is currently implemented in explicit code paths all the way down to the engine. However, conceptually this is no different than any other versioned operation - instead of requiring a document is on a specific version, we require it to be deleted (or non-existent). This PR removes Engine.Create in favor of a slight extension in the VersionType logic.

There are however a couple of side effects:
- DocumentAlreadyExistsException is removed and VersionConflictException is used instead (with an improved error message)
- Update will reject version parameters if the upsert option is used (it doesn't compute anyway).
- Translog.Create is also removed infavor of Translog.Index (that's OK because their binary format was the same, so we can just read Translog.Index of the translog file)

Closes #13955
2015-10-07 12:37:34 +02:00
Greg Marzouka 9200c446a1 Merge pull request #13946 from elastic/fix/indices-get-restspec
Add options for indices.get feature
2015-10-06 14:28:11 -04:00
javanna 1915c74e93 Merge branch 'master' into feature/search-request-refactoring 2015-10-06 16:21:58 +02:00
javanna f89de33548 Merge branch 'master' into feature/search-request-refactoring 2015-10-06 14:28:31 +02:00
Martijn Laarman fd1cd60e78 Fix indices.get_field_mapping rest tests
After `field` has been renamed to `fields` in #13902
2015-10-06 12:40:08 +02:00
Martijn Laarman a0fa83e3df Merge pull request #13678 from elastic/fix/cluster-health-api
In cluster health REST spec, {index} can be one or many indices and should be typed to list.
2015-10-06 11:28:47 +02:00
Martijn Laarman 47871eb854 Merge pull request #13711 from elastic/fix/api-delete-search-template
id route value is required
2015-10-06 11:28:35 +02:00
Martijn Laarman c12cb0b60a Merge pull request #13889 from elastic/fix/indices-get-template-restspec
indices get template accepts a list of names
2015-10-06 11:28:13 +02:00
Martijn Laarman 536ee2b5eb Merge pull request #13900 from elastic/fix/indices-open-restspec
indices.open takes a list of indices
2015-10-06 11:27:52 +02:00
Martijn Laarman fd6652f59f Merge pull request #13901 from elastic/fix/indices-close-restspec
indices.close takes a list of indices
2015-10-06 11:27:12 +02:00
Martijn Laarman e4049ccd8e Merge pull request #13902 from elastic/fix/get-field-mapping-restspec
Get field mapping documented fields as field
2015-10-06 11:26:46 +02:00
Honza Král f2fa279af1 [test] remove index= from catch: /regex/ in date math tests
This breaks on some clients and is not required for the actual test
2015-10-05 20:52:52 +02:00
Greg Marzouka eef34475fd Add options for indices.get feature
As described here https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html?q=get%20index#_filtering_index_information
2015-10-05 12:48:30 -04:00
Colin Goodheart-Smithe ec51e8e31d Merge branch 'master' into feature/search-request-refactoring 2015-10-05 13:31:52 +01:00
Martijn Laarman 6c7e135b49 Get field mapping documented fields as field 2015-10-02 12:54:07 +02:00
Martijn Laarman 80b5a58835 indices.close takes a list of indices
not single index
2015-10-02 12:30:40 +02:00
Martijn Laarman c3f25a0713 indices.open takes a list of indices
not single index
2015-10-02 12:28:53 +02:00
Martijn Laarman 56cbbc4710 indices get template accepts a list of names
not just string
2015-10-01 20:32:22 +02:00
André Carvalho 03c6e8e1cb Adds disk used by indices to _cat/allocation
Sets Store flag on request
2015-10-01 08:16:58 -03:00
Colin Goodheart-Smithe a21238beda Merge branch 'master' into feature/search-request-refactoring
# Conflicts:
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/ChildQuerySearchTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/IndexedScriptTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SearchFieldsTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SearchStatsTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SearchTimeoutTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SimpleSortTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/script/groovy/GroovyScriptTests.java
#	plugins/lang-groovy/src/test/java/org/elasticsearch/script/groovy/GroovySecurityTests.java
2015-09-29 14:40:27 +02:00
Robert Muir e0d42739dd Factor groovy out of core into lang-groovy 2015-09-28 20:17:45 -04:00
Colin Goodheart-Smithe 44722700b0 Merge branch 'feature/query-refactoring' into feature/search-request-refactoring
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/fetch/FetchSubPhasePluginIT.java
2015-09-25 13:04:50 +01:00
Martijn van Groningen 93ad696966 Index name expressions should not be broken up
Closes #13665
2015-09-25 11:28:52 +02:00
Colin Goodheart-Smithe 9f08d48d34 Merge branch 'feature/query-refactoring' into feature/search-request-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/index/query/IndexQueryParserService.java
#	core/src/test/java/org/elasticsearch/script/IndexedScriptIT.java
#	core/src/test/java/org/elasticsearch/script/OnDiskScriptIT.java
2015-09-25 10:14:53 +01:00
Colin Goodheart-Smithe 2dce527b81 rest test fixes (hopefully) 2015-09-23 21:15:13 +01:00
Colin Goodheart-Smithe 4996b07205 update rest tests to expect default boost to be returned in warmer responses 2015-09-23 21:02:55 +01:00
Simon Willnauer 6c335f4a47 Parse source, extra-source and template-source on the coordinating node
This commit removes all the opaque bytes for extra_source and template_source.
Instead source and extra_source etc. are represented as SearchSourceBuilder which can
in-place be modified and is updated with the content of the request parameters.
Template Source is parsed and evaluated which in-turn replaces the actual source.
2015-09-23 12:24:07 +02:00
Robert Muir 689af1a6d6 Factor expressions scripts out to lang-expression plugin 2015-09-22 20:33:47 -04:00
Martijn Laarman 2ee82fc513 id route value is required
id route value is required
2015-09-22 13:08:37 +02:00
Martijn Laarman fcd8606f98 {index} can be one or many indices and should be typed to list.
{index} can be one or many indices and should be typed to list.
2015-09-21 17:38:10 +02:00
Ryan Ernst 18c519145d Remove unnecessary copies of license and notice files
We moved a lot of repositories into elasticsearch, but in their new
location they retained their LICENSE.txt and NOTICE.txt files. These are
all the same, and having the license and notice and the root of the
repository should be sufficient.
2015-09-18 17:48:30 -07:00
Nik Everett 820b721788 [test] Rename start to verify_index in cat test
This was renamed in other places but not here.

Closes #13489
2015-09-10 16:58:28 -04:00
Nik Everett 56c3471851 Fix test for _cat/nodeattrs
Adds a node attribute to all test runs and uses the attribute to test
`_cat/nodeattrs`.

Note that its quite possible create an impressively slow regex while doing
this and you have to be careful. See comment in commit for more if curious.

Closes #12558
2015-09-10 10:50:51 -04:00
Nik Everett 3839d15ea0 Merge branch 'pr/13130' 2015-09-10 10:13:01 -04:00
Nik Everett 2574b5e7f4 [test] Fix help test for _cat/shards
We added a new field and the help test didn't expect it to be there.
2015-09-10 09:35:03 -04:00
Martijn Laarman 8e7be8b552 part types are not string but list on the _search_shards endpoint 2015-09-10 12:02:11 +02:00
Martijn Laarman aaf7116bfb synced flush incorrectly documented url parameters as path parameters 2015-09-10 12:00:06 +02:00
Adrien Grand 0c26e7cd83 Remove the scan and count search types.
These search types have been deprecated in 2.1 and 2.0 respectively, and will
be removed in 3.0.
2015-09-07 15:18:45 +02:00
Simon Willnauer 796701d52e Move version to 3.0.0-SNAPSHOT 2015-09-03 10:43:28 +02:00
Isabel Drost-Fromm 8cd86a615a Adds template support to _msearch resource
Much like we already do with search this adds templating support to the _msearch resource.

Closes #10885
2015-09-01 11:54:43 +02:00
Nik Everett c180defb10 [CAT] Default verbose to false
Closes #13156
2015-08-28 11:15:44 -04:00
Nik Everett 9eb684da51 Default detect_noop to true
detect_noop is pretty cheap and noop updates compartively expensive so this
feels like a sensible default.

Also had to do some testing and documentation around how _ttl works with
detect_noop.

Closes #11282
2015-08-27 10:34:18 -04:00
Clinton Gormley 33e04083a7 Corrected some typos in the REST spec 2015-08-27 11:28:27 +02:00
Clinton Gormley a254b2da29 REST spec: Added update_all_types to indices.put_mapping and indices.create
Closes #12840
2015-08-26 15:33:23 +02:00
Spencer 56d7cb8bd8 Update indices.shard_stores.json
Correct documentation url
2015-08-25 13:21:04 -07:00
David Pilato c189c1a1ec [maven] also rename artifactId for dev-tools and rest-api-spec 2015-08-18 13:41:12 +02:00
David Pilato 0014728a0d [maven] rename maven names for parent, dev-tools and rest spec
Follow up for previous commit:

```
[INFO] Build Tools and Resources .......................... SUCCESS [  0.158 s]
[INFO] Rest API Specification ............................. SUCCESS [  0.002 s]
[INFO] Elasticsearch Parent POM ........................... SUCCESS [  0.122 s]
```
2015-08-18 13:38:49 +02:00
Alexander Reelsen d1c93fb573 Release: Remove aws-maven plugin/improve release docs
In order to have consistent deploys across several repositories,
we should deploy to sonatype first, then mirror those contents,
and then upload to s3.

This means, the aws wagon is not needed anymore.
2015-08-17 15:39:22 +02:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Alexander Reelsen e869c87119 Build: Adding aws-maven extension to dev-tools and rest-api-spec 2015-08-14 17:30:00 +02:00
Simon Willnauer c1405d26b5 fixtypo 2015-08-14 15:27:23 +02:00
Simon Willnauer dd1b0d84a6 Also deploy top level artifacts to S3
This commit adds the S3 wagon release profile also to dev-tools
and rest-api-spec and makes the actual repository path / bucket
configurable.
2015-08-14 15:21:06 +02:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
Simon Willnauer fe3179d9cc Return `408 REQUEST_TIMEOUT` if `_cluster/health` times out
Today we return `200 OK` which is misleading since we really didn't
produce a valid response / didn't wait long enough.
2015-08-12 17:51:20 +02:00
Colin Goodheart-Smithe 6abc69c488 Packaging: Makes sure all POMs contain a description
Adds an explicit description the RPM package so it doesn't inherit the description from the POM.

Closes #12550

Also, modified descriptions for deb and rpm packages to be the same and to reference the documentation rather than listing features that are out of date.
2015-08-10 15:21:24 +01:00
Igor Motov f71c9a25a1 Check for incompatible mappings while upgrading old indices
Conflicting mappings that were allowed before v2.0 can cause runaway shard failures on upgrade. This commit adds a check that prevents a cluster from starting if it contains such indices as well as restoring such indices from a snapshot into already running cluster.

Closes #11857
2015-08-04 18:13:29 -06:00
Robert Muir 6f9a067197 Change master branch back to 2.0-beta1 2015-08-04 15:38:21 -04:00
Simon Willnauer 2b2657279d Use UTC instead of default timezone for creation date in CAT endpoint
this change was added recently which uses default timezone for the creation
date on CAT endpoints. We should be consistent and use UTC across the board.
This commit adds #getDefaultTimzone() to forbidden API and fixes the REST tests.

Relates to #11688
2015-08-04 14:38:23 +02:00
Simon Willnauer 6753f7f03e Cut over master to 2.0.0-SNAPSHOT 2015-08-04 10:54:12 +02:00
Nik Everett 2078100390 [TESTS] cat indices times for UTC - timezones 2015-08-03 11:46:27 -04:00
szroland 94f8b8f2cf remove extra space in test regexp 2015-08-02 21:52:00 +02:00
szroland 9bb904b155 testing output, both with default field names and shorthand aliases 2015-08-02 20:53:03 +02:00
Alexander Reelsen 291bac0f04 Tests: Silencing failing node attrs REST test 2015-07-30 17:48:01 +02:00
Dave Parfitt f209809716 Add _cat/nodeattrs API
This provides a _cat/nodeattrs API call, which presents
custom node attributes in a denormalized table.

Closes #8000
2015-07-29 16:11:18 -04:00
Martijn van Groningen a14913f7b6 Left over from the `query_cache` to `request_cache` rename. 2015-07-27 13:28:15 +02:00
Jason Tedor 8637784a2a Add scroll stats to cat API
Adds scroll stats at the node, shard and index levels to the cat API.

Closes #12330
2015-07-19 13:41:46 -04:00
Areek Zillur 99363bd35d [TEST] fix test bug 2015-07-16 18:57:04 -04:00
Areek Zillur 7a21d846bb A new `_shard_stores` API provides store information for shard copies of indices.
Store information reports on which nodes shard copies exist, the shard
copy version, indicating how recent they are, and any exceptions
encountered while opening the shard index or from earlier engine failure.

closes #10952
2015-07-16 18:28:07 -04:00
Clinton Gormley d4b94a9be2 Revert "REST: Fixed msearch and mpercolate structured exception tests"
This reverts commit f20b730eca.
2015-07-16 17:06:45 +02:00
Clinton Gormley f20b730eca REST: Fixed msearch and mpercolate structured exception tests 2015-07-16 15:27:07 +02:00
Simon Willnauer def08bd594 Render structured exceptions in mget / mpercolate
Instead of rendering only the exception message this commit
adds structured exception rendering to mget and mpercolate
2015-07-14 17:52:28 +02:00
Simon Willnauer 7db293c616 Generify Index and Shard exceptions
Today we have a intermediate hierarchy for shard and index exceptions
which makes it hard to introduce generic exceptions like ResourceNotFoundException
intoduced in this commit. This commit breaks up the hierarchy by adding index and shard
as a special internal header that gets rendered for every exception that fills that header.
This commit removes dedicated exceptions like `IndexMissingException` or
`IndexShardMissingException` in favour of `ResourceNotFoundException`
2015-07-14 16:31:49 +02:00
Simon Willnauer fbafd14f23 [TEST] Don't use replicas to stabelize assertion.
This test asserts that the first recovery result is of type SNAPSHOT.
This assertion might not be true depending on the rendering order if
a replica is recovered quick enough. This commit disables replicas and
their recovery since it's not the purpose of this test.
2015-07-14 10:32:20 +02:00
uboness b40186652c updated the elasticsearch versioning format
Moving to from `X.Y.Z.beta1`/`X.Y.Z.RC1` to `X.Y.Z-beta1`/`X.Y.Z-rc1`
2015-07-13 20:26:37 +02:00
Jack Conradson 82b112b911 Minor fix to a failing REST test due to the prior check-in. 2015-07-10 19:30:28 -07:00
Simon Willnauer e0708813a9 Make 2.0.0.beta1-SNAPSHOT the current version.
Today everything is tight to having the next version as the latest.
In order to work towards 2.0.0.beta1 we need to fix all the usage of
2.0.0-SNAPSHOT to reflect the version we will release soon.
Usually we do this on the release branch but to simplify things I wanna
keep this on master for now and move to 2.1.0-SNAPSHOT on master once
we created a 2.0 branch.

Closes #12148
2015-07-09 21:24:32 +02:00
Tanguy Leroux 26df78a90a Fix REST test for cat.nodes (Windows load average OS stats return -1.00) 2015-07-09 19:52:52 +02:00
Igor Motov f303a1d4eb Snapshot info should contain version of elasticsearch that created the snapshot
This information was stored with the snapshot but wasn't available on the interface. Knowing the version of elasticsearch that created the snapshot can be useful to determine the minimal version of the cluster that is required in order to restore this snapshot.

Closes #11980
2015-07-09 12:02:33 -04:00
Lee Hinman 2213a5aa81 Revert "[TEST] Add REST awaitsFix for bulk fields test"
This reverts commit 799e801980.
2015-07-08 12:00:37 -06:00
Lee Hinman 799e801980 [TEST] Add REST awaitsFix for bulk fields test
Relates to #12133
2015-07-08 11:59:34 -06:00
Adrien Grand ccbea2e5d2 Fix parsing of the `fields` parameter of bulk requests. 2015-07-08 19:55:24 +02:00
Jason Tedor b61709c716 Add support for retrieving fields in bulk updates
This commit adds support to retrieve fields when using the bulk update API. This functionality was previously available for the update API
but not for the bulk update API.

Closes #11527
2015-07-08 12:06:20 -04:00
Tanguy Leroux 30892c4129 Remove network stats & info 2015-07-07 21:16:42 +02:00
Igor Motov 177180ac94 Fix snapshot-related REST integration tests 2015-07-06 14:16:28 -04:00
David Pilato e7a6b51bab [maven] change groupId / artifactId
When we generate our project, we can get something like:

```
├── dev-tools
├── elasticsearch
├── elasticsearch-parent
├── elasticsearch-plugin
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   └── elasticsearch-lang-python
├── rest-api-spec
└── securemock
```

I propose here to use a common naming for artifacts: start always with `elasticsearch-`.
Also, move `elasticsearch-plugin` to `org.elasticsearch.plugin` groupId.

So we could have:

```
├── elasticsearch
├── elasticsearch-dev-tools
├── elasticsearch-parent
├── elasticsearch-rest-api-spec
├── elasticsearch-securemock
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   ├── elasticsearch-lang-python
│   └── elasticsearch-plugin
```
2015-07-06 17:17:07 +02:00
Ryan Ernst 410704fd18 fix rest tests to not depend on short name access of mappings 2015-07-02 11:37:18 -07:00
szroland 68d658a30f Recovery: Fix wrong reused file bytes in Recovery API reports
Simple snapshot.restore test case that also exposes the bug mentioned in #11876

Fix #11876
Closes #11965
2015-07-01 09:08:53 +02:00
Martijn van Groningen ef9d70b9b3 field stats: added index constraints
Field stats index constraints allows to omit all field stats for indices that don't match with the constraint. An index
constraint can exclude indices' field stats based on the `min_value` and `max_value` statistic. This option is only
useful if the `level` option is set to `indices`.

For example index constraints can be useful to find out the min and max value of a particular property of your data in
a time based scenario. The following request only returns field stats for the `answer_count` property for indices
holding questions created in the year 2014:

curl -XPOST 'http://localhost:9200/_field_stats?level=indices' -d '{
   "fields" : ["answer_count"] <1>
   "index_constraints" : { <2>
      "creation_date" : { <3>
         "min_value" : { <4>
            "gte" : "2014-01-01T00:00:00.000Z",
         },
         "max_value" : {
            "lt" : "2015-01-01T00:00:00.000Z"
         }
      }
   }
}'

Closes #11187
2015-07-01 08:47:03 +02:00
Colin Goodheart-Smithe a847dd267c [TEST] Fix location of render_search_template.json 2015-06-30 17:45:43 +01:00
Colin Goodheart-Smithe d9ab3cba77 Search Templates: Adds API endpoint to render search templates as a response
Closes #6821
2015-06-30 16:57:23 +01:00
szroland e2d76aaa20 remove _create from POST to match PUT path 2015-06-29 21:55:52 +02:00
Masaru Hasegawa 80d0f1e5ea Fix RecoveryState timestamps
Use System.currentTimeMillis() for wall clock time and use System.nanoTime() for delta.
Fixes #11870.
2015-06-29 19:24:23 +09:00
Adrien Grand 38f5cc236a Rename caches.
In order to be more consistent with what they do, the query cache has been
renamed to request cache and the filter cache has been renamed to query
cache.

A known issue is that package/logger names do no longer match settings names,
please speak up if you think this is an issue.

Here are the settings for which I kept backward compatibility. Note that they
are a bit different from what was discussed on #11569 but putting `cache` before
the name of what is cached has the benefit of making these settings consistent
with the fielddata cache whose size is configured by
`indices.fielddata.cache.size`:
 * index.cache.query.enable -> index.requests.cache.enable
 * indices.cache.query.size -> indices.requests.cache.size
 * indices.cache.filter.size -> indices.queries.cache.size

Close #11569
2015-06-29 10:15:27 +02:00
Simon Willnauer dc67bd0021 Render strucutred exception in multi search
MultiMatch still only returns the exception message but should return the
actual exception and render it in a structured fashion
2015-06-24 16:21:41 +02:00
Simon Willnauer fcdcce3bba Consolidate shard level abstractions
This commit consolidates several abstractions on the shard level in
ordinary classes not managed by the shard level guice injector.

Several classes have been collapsed into IndexShard and IndexShardGatewayService
was cleaned up to be more lightweight and self-contained. It has also been moved into
the index.shard package and it's operation is renamed from recovery from "gateway" to recovery
from "store" or "shard_store".

Closes #11847
2015-06-24 15:18:04 +02:00
Ryan Ernst 12e7cbe92b Mappings: Lockdown _timestamp
This is a follow up to #8143 and #6730 for _timestamp. It removes
support for `path`, as well as any field type settings, and
enables docvalues for _timestamp, for 2.0.  Users who need to
adjust these settings can use a date field.
2015-06-22 10:21:03 -07:00
Alexander Reelsen 88f8d58c8b Cluster Health: Add max wait time for pending task and active shard percentage
In order to get a quick overview using by simply checking the cluster state
and its corresponding cat API, the following two attributes have been added
to the cluster health response:

* task max waiting time, the time value of the first task of the
  queue and how long it has been waiting
* active shards percent: The percentage of the number of shards that are in
  initializing state

This makes the cluster health API handy to check, when a fully restarted
cluster is back up and running.

Closes #10805
2015-06-22 15:04:47 +02:00
Colin Goodheart-Smithe 772d0cc6e7 Build: Make rest-spec-api a project so eclipse build works
The change makes rest-spec-api a project in the same way as we build dev-tools. it packages the tests and api in a bundle using the maven-remote-resources-plugin and uses the same plugin in the plugins and core pom to unpack the rest-api-spec into the target directory and references the rest tests there in the test resources.

The main stimulus for this change is that for those using Eclipse the current build does not work. After running `mvn eclipse:eclipse` the Eclipse IDE errors because the rest-api-spec is outside of the project scope, meaning that every time the command is run (required whenever any dependencies change), the class path of all the projects has to be manually fixed.
2015-06-22 11:41:44 +01:00
Igor Motov 36da42c93b Show human readable Elasticsearch version that created index and date when index was created
By setting human parameter to true, it's now possible to see human readable versions of Elasticsearch that created and updated the index as well as the date when the index was created.

Closes #11484
2015-06-17 18:52:01 -04:00
Alex Ksikes 3f6dae1a73 More Like This: renamed `ignore_like` to `unlike`
This changes the parameter name `ignore_like` to the more user friendly name
`unlike`. This later feature generates a query from the terms in `A` but not
from the terms in `B`. This translates to a result set which is like `A` but
unlike `B`. We could have further negatively boosted any documents that have
some `B`, but these documents already do not receive any contribution from
having `B`, and would therefore negatively compete with documents having `A`.

Closes #11117
2015-06-17 17:18:50 -05:00
Shay Banon 4e0d65453f Add Unassigned meta data
Unassigned meta includes additional information as to why a shard is unassigned, this is especially handy when a shard moves to unassigned due to node leaving or shard failure.

The additional data is provided as part of the cluster state, and as part of `_cat/shards` API.

The additional meta includes the timestamp that the shard has moved to unassigned, allowing us in the future to build functionality such as delay allocation due to node leaving until a copy of the shard is found.
closes #11653
2015-06-16 17:01:48 +02:00
Boaz Leskes e559eafa19 Revert "Test: fix removal of e validate_query rewrite YAML test"
This reverts commit e461852548.
2015-06-15 21:57:05 +02:00
Boaz Leskes e461852548 Test: fix removal of e validate_query rewrite YAML test 2015-06-15 20:29:06 +02:00
gmarz 3b93653863 [TEST] Remove validate_query rewrite YAML test 2015-06-15 12:52:06 -04:00
gmarz 8dd7dae5b9 [SPEC] Add `rewrite` query parameter to the `indices.validate_query` API 2015-06-15 11:13:46 -04:00
Simon Willnauer 43ce27d3bd [TEST] Shards can also be RELOCATING 2015-06-15 10:07:40 +02:00
Honza Král 09b5f90779 [API] correct link for flush_synced docs 2015-06-09 21:29:02 +02:00
Nirmal Chidambaram 931b9f9c74 Filtered out non data-nodes in relevant cat api
Closes #9214
Closes #9287
2015-06-08 16:05:42 +02:00
Boaz Leskes e359698605 test: reduce replicas for indices synced flush test
We have to make sure all shards are started to know the synced flush will hit them all. Shards that are still initializing during the sync flush may be missed and confuse the stats call
2015-06-06 21:10:25 +02:00
Adrien Grand 7c698146f5 Rest: Add all meta fields to the top level json document.
Some of our meta fields (such as _id, _version, ...) are returned as top-level
properties of the json document, while other properties (_timestamp, _routing,
...) are returned under `fields`. This commit makes all meta fields returned
as top-level properties.

So eg. `GET test/test/1?fields=_timestamp,foo` would now return

```json
{
   "_index": "test",
   "_type": "test",
   "_id": "1",
   "_version": 1,
   "_timestamp": 10000000,
   "found": true,
   "fields": {
     "foo": [ "bar" ]
   }
}
```

while it used to return

```json
{
   "_index": "test",
   "_type": "test",
   "_id": "1",
   "_version": 1,
   "found": true,
   "fields": {
     "_timestamp": 10000000,
     "foo": [ "bar" ]
   }
}
```
2015-06-04 23:42:17 +02:00
Michael McCandless e1197dfea9 Merge branch 'master' into require_units
Conflicts:
	src/main/java/org/elasticsearch/action/bulk/BulkRequest.java
	src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexUpgradeService.java
	src/main/java/org/elasticsearch/node/internal/InternalSettingsPreparer.java
	src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java
2015-06-04 13:59:10 -04:00
Britta Weber d23449ee85 [test] fix name in synced flush test 2015-05-29 13:48:37 +02:00
Clinton Gormley 18c8394fbd REST spec: Renamed indices.flush.synced to indices.flush_synced 2015-05-29 12:30:43 +02:00
Britta Weber 87a0c76e9c Merge remote-tracking branch 'boaz/index_seal_to_flush_sync' 2015-05-29 10:31:03 +02:00
Igor Motov c695f35bca Tests: make REST upgrade test more reliable
Make sure that all shards are started to avoid flush conflicts.
2015-05-28 12:37:52 -10:00
Igor Motov 6980286ba4 Core: fix upgrade response serialization 2015-05-28 09:46:46 -10:00
Alexander Reelsen 38639074b4 Testing: Ensure cat API REST tests are unaffected by randomization
The wildcard cat API REST tests relied on bulk.max and bulk.min in
the thread_pool response. However due to the thread pool types being
randomized in InternalTestCluster, the min/max values were not guaranteed
to exist (the cached thread pool type is unbounded and thus does not have a
max value).

In order to prevent this, the test has been removed and now the cat
nodes test is used for wildcard testing, which always returns stats
about the heap.
2015-05-28 16:24:41 +02:00
Clinton Gormley cbb7b633f6 REST tests: Fixed bad YAML in search/10_source_filtering 2015-05-27 19:35:00 +02:00
Alexander Reelsen 9d5e789508 Cat API: Do not rely on hashmap for sorted entries
The tests for the recently added added wildcard feature were
relying on order of the hashmap being used, which could be
different.

The implementation now ensures, that the header fields are
parsed in the order they have been added.
2015-05-27 17:46:22 +02:00
markharwood 57653df713 Search fix: fielddata_fields query string parameter was ignored.
The RestSearchAction did not parse the fielddata_fields parameter. Added test case and missing parser code.

Closes #11025
2015-05-27 15:30:10 +01:00
Alexander Reelsen fc224a0de8 Cat API: Add wildcard support for header names
This adds wildcard support (simple regexes) for specifying header names.
Aliases are supported as well.

Closes #10811
2015-05-27 16:09:31 +02:00
javanna 44fe99a3a8 [TEST] make filter_path a default parameter in java rest runner
Closes #11351
2015-05-26 15:34:45 +02:00
Tanguy Leroux ce63590bd6 API: Add response filtering with filter_path parameter
This change adds a new "filter_path" parameter that can be used to filter and reduce the responses returned by the REST API of elasticsearch.

For example, returning only the shards that failed to be optimized:
```
curl -XPOST 'localhost:9200/beer/_optimize?filter_path=_shards.failed'
{"_shards":{"failed":0}}%
```

It supports multiple filters (separated by a comma):
```
curl -XGET 'localhost:9200/_mapping?pretty&filter_path=*.mappings.*.properties.name,*.mappings.*.properties.title'
```

It also supports the YAML response format. Here it returns only the `_id` field of a newly indexed document:
```
curl -XPOST 'localhost:9200/library/book?filter_path=_id' -d '---hello:\n  world: 1\n'
---
_id: "AU0j64-b-stVfkvus5-A"
```

It also supports wildcards. Here it returns only the host name of every nodes in the cluster:
```
curl -XGET 'http://localhost:9200/_nodes/stats?filter_path=nodes.*.host*'
{"nodes":{"lvJHed8uQQu4brS-SXKsNA":{"host":"portable"}}}
```

And "**" can be used to include sub fields without knowing the exact path. Here it returns only the Lucene version of every segment:
```
curl 'http://localhost:9200/_segments?pretty&filter_path=indices.**.version'
{
  "indices" : {
    "beer" : {
      "shards" : {
        "0" : [ {
          "segments" : {
            "_0" : {
              "version" : "5.2.0"
            },
            "_1" : {
              "version" : "5.2.0"
            }
          }
        } ]
      }
    }
  }
}
```

Note that elasticsearch sometimes returns directly the raw value of a field, like the _source field. If you want to filter _source fields, you should consider combining the already existing _source parameter (see Get API for more details) with the filter_path parameter like this:

```
curl -XGET 'localhost:9200/_search?pretty&filter_path=hits.hits._source&_source=title'
{
  "hits" : {
    "hits" : [ {
      "_source":{"title":"Book #2"}
    }, {
      "_source":{"title":"Book #1"}
    }, {
      "_source":{"title":"Book #3"}
    } ]
  }
}
```
2015-05-26 13:51:04 +02:00
Boaz Leskes b376a3fbfb Move index sealing terminology to synced flush
#10032 introduced the notion of sealing an index by marking it with a special read only marker, allowing for a couple of optimization to happen. The most important one was to speed up recoveries of shards where we know nothing has changed since they were online by skipping the file based sync phase. During the implementation we came up with a light notion which achieves the same recovery benefits but without the read only aspects which we dubbed synced flush. The fact that it was light weight and didn't put the index in read only mode, allowed us to do it automatically in the background which has great advantage. However we also felt the need to allow users to manually trigger this operation.

 The implementation at #11179 added the sync flush internal logic and the manual (rest) rest API. The name of the API was modeled after the sealing terminology which may end up being confusing. This commit changes the API name to match the internal synced flush naming, namely `{index}/_flush/synced'.

  On top of that it contains a couple other changes:
   - Remove all java client API. This feature is not supposed to be called programtically by applications but rather by admins.
   - Improve rest responses making structure similar to other (flush) API
   - Change IndexShard#getOperationsCount to exclude the internal +1 on open shard . it's confusing to get 1 while there are actually no ongoing operations
   - Some minor other clean ups
2015-05-25 22:32:32 +03:00
Michael McCandless 5aaebb6d83 add missing units to tests 2015-05-25 06:48:34 -04:00
Clinton Gormley 409e4e5f73 REST test: Fixed index-seal test
Fixed bad YAML, and changed to wait for yellow instead of green, because
REST tests usually run on a single node
2015-05-20 13:46:43 +02:00
Simon Willnauer 9d2852f0ab Merge branch 'master' into feature/synced_flush
Conflicts:
	src/main/java/org/elasticsearch/index/engine/InternalEngine.java
	src/main/java/org/elasticsearch/index/shard/IndexShard.java
	src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java
	src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
2015-05-19 12:16:22 +02:00
Adrien Grand 2c241e8a36 Mappings: Remove the `ignore_conflicts` option.
Mappings conflicts should not be ignored. If I read the history correctly, this
option was added when a mapping update to an existing field was considered a
conflict, even if the new mapping was exactly the same. Now that mapping updates
are smart enough to detect conflicting options, we don't need an option to
ignore conflicts.
2015-05-18 15:28:23 +02:00
Martijn van Groningen 719252a138 Merge pull request #11183 from martijnvg/parent-child/remove_id_cache_from_stats_and_clear_cache_apis
Removed `id_cache` from stats and cat apis.
2015-05-15 14:39:35 +02:00
Martijn van Groningen ece18f162e Removed `id_cache` from stats and cat apis.
Also removed the `id_cache` option from the clear cache api.

Closes #5269
2015-05-15 14:06:18 +02:00
Jun Ohtani 3a1a4d3e89 Analysis: Add multi-valued text support
Add support array text as a multi-valued for AnalyzeRequestBuilder
Add support array text as a multi-valued for Analyze REST API
Add docs

Closes #3023
2015-05-15 20:01:10 +09:00
Britta Weber c628d67f9e api for synced flush
closes #11098
2015-05-13 14:45:24 +02:00
javanna d7e585ca4f REST: Unify query_string parameters parsing
There currently are small differences between search api and count, exists, validate query, explain api when it comes to reading query_string parameters.  `analyze_wildcard`, `lowercase_expanded_terms` and `lenient` are only read by the search api and ignored by all other mentioned apis. Unified code to fix this and make sure it doesn't happen again. Also shared some code when it comes to printing out the query as part of SearchSourceBuilder conversion to ToXContent.

Extended REST spec to include all the supported params (some that were already supported weren't listed), and added REST tests (also some basic tests for count and search_exists which weren't tested at all).

Closes #11057
2015-05-11 11:32:59 +02:00
Andrew Selden c953e99324 Merge pull request #10864 from aleph-zero/issues/9606
Remove (dfs_)query_and_fetch from the REST API
2015-05-07 12:51:28 -07:00
Alex Ksikes ec4f12f9ef More Like This: removal of the MLT API
Removes the More Like This API, users should now use the More Like This query.
The MLT API tests were converted to their query equivalent. Also some clean
ups in MLT tests.

Closes #10736
Closes #11003
2015-05-06 18:11:11 +02:00
Adrien Grand b72f27a410 Core: Cut over to the Lucene filter cache.
This removes Elasticsearch's filter cache and uses Lucene's instead. It has some
implications:
 - custom cache keys (`_cache_key`) are unsupported
 - decisions are made internally and can't be overridden by users ('_cache`)
 - not only filters can be cached but also all queries that do not need scores
 - parent/child queries can now be cached, however cached entries are only
   valid for the current top-level reader so in practice it will likely only
   be used on read-only indices
 - the cache deduplicates filters, which plays nicer with large keys (eg. `terms`)
 - better stats: we already had ram usage and evictions, but now also hit count,
   miss count, lookup count, number of cached doc id sets and current number of
   doc id sets in the cache
 - dynamically changing the filter cache size is not supported anymore

Internally, an important change is that it removes the NoCacheFilter infrastructure
in favour of making Query.rewrite specializing the query for the current reader so
that it will only be cached on this reader (look for IndexCacheableQuery).

Note that consuming filters with the query API (createWeight/scorer) instead of
the filter API (getDocIdSet) is important for parent/child queries because
otherwise a QueryWrapperFilter(ParentQuery) would run the wrapped query per
segment while relations might be cross segments.
2015-05-04 09:02:15 +02:00
Honza Král 28e5a649b4 [API] remove global parameter source from individual APIs
Same way we don't define pretty anywhere, we shouldn't have source
2015-04-29 14:01:47 +02:00
aleph-zero 89542facb3 Remove (dfs_)query_and_fetch from the REST API
Remove the ability to specify search type ‘query_and_fetch’ and
‘df_query_and_fetch’ from the REST API.

- Adds REST tests
- Updates REST API spec to remove ‘query_and_fetch’ and
‘df_query_and_fetch’ as options
- Removes documentation for these options

Closes #9606
2015-04-28 15:27:59 -07:00
Michael McCandless cf2fb4ed0f Remove core delete-by-query implementation, to be replaced with a plugin
The current implementation is dangerous: it unexpectedly refreshes,
which can quickly cause an unhealthy index (segment explosion).  It
can also delete different documents on primary vs replicas, causing
inconsistent replicas.

For 2.0 we will replace this with an optional plugin that does a
scan/scroll search and then issues bulk delete requests.

Closes #10859
2015-04-28 16:09:04 -04:00
Simon Willnauer d164526d27 Remove `_shutdown` API
Thsi commit removes the `_shutdown` API entirely without any replacement.
Nodes should be managed from the operating system not via REST APIs
2015-04-27 17:19:36 +02:00
Simon Willnauer 15d58d91f1 [REST] Render REST errors in a structural way
This commit adds support for structural errors / failures / exceptions
on the elasticsearch REST layer. Exceptions are rendering with at least
a `type` and a `reason` corresponding to the exception name and the message.
Some expcetions like the ones associated with an index or a shard will have
additional information about the index the exception was triggered on or the
shard respectivly.

Each rendered response will also contain a list of root causes which is a list
of distinct shard level errors returned for the request. Root causes are the lowest
level elasticsearch exception found per shard response and are intended to be displayed
to the user to indicate the soruce of the exception.

Shard level response are by-default grouped by their type and reason to reduce the amount
of duplicates retunred. Yet, the same exception retunred from different indices will not be
grouped.

Closes #3303
2015-04-24 09:36:10 +02:00
tlrx adc0807c68 Internal: Add METADATA_READ and METADATA_WRITE blocks
This commit splits the current ClusterBlockLevel.METADATA into two disctins ClusterBlockLevel.METADATA_READ and ClusterBlockLevel.METADATA_WRITE blocks. It allows to make a distinction between
an operation that modifies the index or cluster metadata and an operation that does not change any metadata.

Before this commit, many operations where blocked when the cluster was read-only: Cluster Stats, Get Mappings, Get Snapshot, Get Index Settings, etc. Now those operations are allowed even when
the cluster or the index is read-only.

Related to #8102, #2833

Closes #3703
Closes #5855
Closes #10521
Closes #10522
2015-04-23 15:13:57 +02:00
Martijn van Groningen 5705537ecf Added field stats api
The field stats api returns field level statistics such as lowest, highest values and number of documents that have at least one value for a field.

An api like this can be useful to explore a data set you don't know much about. For example you can figure at with the lowest and highest response times are, so that you can create a histogram or range aggregation with sane settings.

This api doesn't run a search to figure this statistics out, but rather use the Lucene index look these statics up (using Terms class in Lucene). So finding out these stats for fields is cheap and quick.

The min/max values are based on the type of the field. So for a numeric field min/max are numbers and date field the min/max date and other fields the min/max are term based.

Closes #10523
2015-04-23 08:52:34 +02:00
Ryan Ernst 2d54738339 Tests: Add shortcut "all" to skip version ranges in rest tests
This was suggested on #10656 as cleaner than " - " to indicate all
versions should be skipped.

closes #10702
2015-04-22 11:40:26 -07:00
Boaz Leskes 1118965aa5 Test: move to ensure yellow in indices.stats/12_level.yaml 2015-04-22 16:10:28 +02:00
Boaz Leskes 91ff3f6963 Test: add ensure green to indices.stats/12_level.yaml
Also changed the stash logger to not log all stashed values under debug (it does trace now) but do dump the stash content upon failure (under info as a XContent)
2015-04-22 15:56:45 +02:00
Jun Ohtani 0955c127c0 Rest: Add json in request body to scroll, clear scroll, and analyze API
Change analyze.asciidoc and scroll.asciidoc
Add json support to Analyze and Scroll, and clear scrollAPI
Add rest-api-spec/test

Closes #5866
2015-04-22 17:53:20 +09:00
Boaz Leskes a1ba339517 Stats: add CommitStats to supply information about the current commit point
Extends ShardStats with commit specific information. We currently expose commit id, generation and the user data map.

The information is also retrievable via the Rest API by using `GET _stats?level=shards`

Closes #10687
2015-04-22 09:33:14 +02:00
Adrien Grand d7abb12100 Replace deprecated filters with equivalent queries.
In Lucene 5.1 lots of filters got deprecated in favour of equivalent queries.
Additionally, random-access to filters is now replaced with approximations on
scorers. This commit
 - replaces the deprecated NumericRangeFilter, PrefixFilter, TermFilter and
   TermsFilter with NumericRangeQuery, PrefixQuery, TermQuery and TermsQuery,
   wrapped in a QueryWrapperFilter
 - replaces XBooleanFilter, AndFilter and OrFilter with a BooleanQuery in a
   QueryWrapperFilter
 - removes DocIdSets.isBroken: the new two-phase iteration API will now help
   execute slow filters efficiently
 - replaces FilterCachingPolicy with QueryCachingPolicy

Close #8960
2015-04-21 15:32:43 +02:00
Adrien Grand 1adf232bb2 Mappings: Validate dynamic mappings updates on the master node.
This commit changes dynamic mappings updates so that they are synchronous on the
entire cluster and their validity is checked by the master node. There are some
important consequences of this commit:
 - a failing index request on a non-existing type does not implicitely create
   the type anymore
 - dynamic mappings updates cannot create inconsistent mappings on different
   shards
 - indexing requests that introduce new fields might induce latency spikes
   because of the overhead to update the mappings on the master node

Close #8688
2015-04-21 11:08:21 +02:00
Ryan Ernst 68f75ea7b6 simplified rest skip range version parsing more, ranges can now be open
ended
2015-04-19 06:50:08 -07:00
Ryan Ernst 06eee11dbb simplify version handling in rest tests, add tests for version utilities 2015-04-18 12:48:00 -07:00
Michael McCandless 399f0ccce9 Core: add only_ancient_segments to upgrade API, so only segments with an old Lucene version are upgraded
This option defaults to false, because it is also important to upgrade
the "merely old" segments since many Lucene improvements happen within
minor releases.

But you can pass true to do the minimal work necessary to upgrade to
the next major Elasticsearch release.

The HTTP GET upgrade request now also breaks out how many bytes of
ancient segments need upgrading.

Closes #10213

Closes #10540

Conflicts:
	dev-tools/create_bwc_index.py
	rest-api-spec/api/indices.upgrade.json
	src/main/java/org/elasticsearch/action/admin/indices/optimize/OptimizeRequest.java
	src/main/java/org/elasticsearch/action/admin/indices/optimize/ShardOptimizeRequest.java
	src/main/java/org/elasticsearch/action/admin/indices/optimize/TransportOptimizeAction.java
	src/main/java/org/elasticsearch/index/engine/InternalEngine.java
	src/test/java/org/elasticsearch/bwcompat/StaticIndexBackwardCompatibilityTest.java
	src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
	src/test/java/org/elasticsearch/rest/action/admin/indices/upgrade/UpgradeReallyOldIndexTest.java
2015-04-16 05:24:33 -04:00
Simon Willnauer b756477fb4 [CORE] Rename START phase into VERIFY_INDEX
we really only optionally run checkindex in this phase and moving
the engine start into translog is move cleaner code wise.

Closes #10570
2015-04-14 09:53:02 +02:00
Spencer Alger 0cd1848e88 Indexed scripts/templates: fix yaml test indentation 2015-04-08 15:36:23 -07:00
javanna acabf2d55a Cluster state REST api: print routing_nodes out only when requested through specific flag
For bacwards compatibility reasons routing_nodes were previously printed out when routing_table was requested, together with the actual routing_table. Now they are printed out only when requests through `routing_nodes` flag.

Relates to #10412
Closes #10486
2015-04-08 16:10:36 +02:00
Leonardo Menezes 5fd9aee16e Cluster state REST api: routing_nodes as an independent metric option
Cluster state api returns both routing_table and routing_nodes sections whenever routing_table is requested. That is pretty much the same info, just grouped differently. This commit allows to differentiate between the two. Yet, routing_table still returns both for bw comp reasons.

Closes #10352
Closes #10412
2015-04-08 15:35:11 +02:00
javanna df875707ec Indexed scripts/templates: return response body when script is not found
Align get indexed scripts and get search template apis to our get api, which returns a response body when the document is not found, with a found boolean flag. Also, return metadata info all the time too.

Closes #7325
Closes #10396
2015-04-08 10:44:11 +02:00
Christoph Büscher 2beda3953d Remove expansion of empty index arguments in RoutingTable
RoutingTables activePrimaryShardsGrouped(), allActiveShardsGrouped() and
allAssignedShardsGrouped() methods treated empty index array input
parameters as meaning "all" indices and expanded to the routing maps
keyset. However, the expansion of index names is now already done in
MetaData#concreteIndices(). Returning an empty index name list here
when a wildcard pattern didn't match any index name could lead to
problems like #9081 because the RoutingTable still expanded this
list of names to "_all". In case of e.g. the recovery endpoint this
could lead to problems.

Closes #9081
Closes #10148
2015-04-02 12:29:47 +02:00
Adrien Grand a608db122d Search: Remove the `count` search type.
This commit brings the benefits of the `count` search type to search requests
that have a `size` of 0:
 - a single round-trip to shards (no fetch phase)
 - ability to use the query cache

Since `count` now provides no benefits over `query_then_fetch`, it has been
deprecated.

Close #7630
2015-03-31 11:31:49 +02:00
Martijn van Groningen 6d1a1b328b Make sure that the parent option on the update request only is delgated to upsert index request.
Closes #4538
2015-03-28 08:53:11 +01:00
Martijn van Groningen 75713f4190 Reverted commit: 20f7be3 2015-03-28 08:53:11 +01:00
Leonardo Menezes 5feaa484b5 Cat api: added node id column to _cat/plugins response
relates to #10226
2015-03-25 12:00:38 +01:00
Leonardo Menezes d086648cec Cat api: added node id column to _cat/segments response
Relates to #10226
2015-03-25 12:00:15 +01:00
Leonardo Menezes bad60cea13 Cat api: added node id column to _cat/shards response
closes #10226
2015-03-25 11:59:35 +01:00
Clinton Gormley 1ee42e93d0 Fix REST test delete/42_missing_parent to catch a RoutingMissingException
Related to #10136
2015-03-25 06:48:05 +02:00
Ryan Ernst 90dfd78267 Remove missed references to delete mapping API
See #10231
2015-03-24 10:13:19 -07:00
Ryan Ernst 693d91e41c Mappings: Remove delete mapping API
Deleting a type from an index is inherently dangerous because
the type can be recreated with new mappings which may conflict
with existing segments still using the old mappings. This
removes the ability to delete a type (similar to how deleting
fields within a type is not allowed, for the same reason).

closes #8877
closes #10231
2015-03-24 09:46:02 -07:00
Honza Král 9c0f655674 [API] changed all the links in json specs to the new site 2015-03-24 17:30:32 +01:00
Tim Schlechter 56117e3f70 [SPEC] Remove duplicated timeout param from bulk REST spec
Closes #10205
2015-03-22 10:26:12 +01:00
javanna 481ef7d3f4 [TEST] add REST test for update index settings api using indices options
Relates to #10030
2015-03-21 11:10:24 +01:00
Simon Willnauer 7257345db9 Revert Benchmark API
The benchmark api is being worked on feature/bench branch and will be merged from there when ready.
2015-03-21 10:36:04 +01:00
Boaz Leskes 8c69535580 Recovery: add throttle stats
This commit adds throttling statistics to the index stats API and to the recovery state.

Closes #10097
2015-03-20 10:13:40 +01:00
Simon Willnauer 1168347b9d [REPLICATION] Remove `async` replication
Closes #10114
2015-03-19 14:44:21 -07:00
Clinton Gormley 6e9eea9b98 Deleted REST test pending shard header
Bug introduced in #10162
2015-03-19 17:57:25 +01:00
Clinton Gormley 25369f0727 Remove async replication from the docs and REST spec
Relates to #10114
2015-03-19 15:34:12 +01:00
Christoph Büscher 5788289a6d [TEST] Remove 'gtelte' feature from yaml tests
Some yaml tests still used the 'gtelte' feature in their skip section.
Since all language clients support this, the feature skip can be
removed.
2015-03-18 15:31:17 -07:00
Boaz Leskes b605184471 Recovery: add total operations to the `_recovery` API
This commit adds the current total number of translog operations to the recovery reporting API. We also expose the recovered / total percentage:

```
"translog": {
   "recovered": 536,
   "total": 986,
   "percent": "54.3%",
   "total_time": "2ms",
   "total_time_in_millis": 2
},
```

Closes #9368
Closes #10042
2015-03-17 07:31:29 -07:00
Lee Hinman 6aec68cd29 Revert "[QUERY] Remove lowercase_expanded_terms and locale options"
This reverts commit d1f7bd97cb.

Ryan pointed out that this needs to work with the multi term query, so
additional analysis and tests should be added.
2015-03-13 13:51:44 -06:00
Lee Hinman d1f7bd97cb [QUERY] Remove lowercase_expanded_terms and locale options
The analysis chain should be used instead of relying on this, as it is
confusing when dealing with different per-field analysers.

The `locale` option was only used for `lowercase_expanded_terms`, which,
once removed, is no longer needed, so it was removed as well.

Fixes #9978
Relates to #9973
2015-03-13 13:17:27 -06:00
Lee Hinman 583c4926c4 Mark shadow replicas with 's' in _cat/shards output
Fixes #9772
2015-03-06 13:21:29 -07:00
javanna 4ad33c30ac [TEST] Work around URI encode limitations in RestClient
We've been relying on URI for url encoding, but it turns out it has some problems. For instance '+' stays as is while it should be encoded to `%2B`. If we go and manually encode query params we have to be careful though not to run into double encoding ('+'=>'%2B'=>'%252B'). The applied solution relies on URI encoding for the url path, but manual url encoding for the query parameters. We prevent URI from double encoding query params by using its single argument constructor that leaves everything as is.

We can also revert back the expression script REST test that revealed this to its original content (which contains an addition).

Closes #9769
Closes #9946
2015-03-03 10:21:23 +01:00
Clinton Gormley 8cba6509e4 REST: Fix script-expressions test to work with Perl
Closes #9931
2015-03-02 10:27:05 +01:00
Clinton Gormley 03d6dae834 REST tests - disable expressions test for now
Java returns floating point but JSON contains integer
2015-02-28 19:31:28 +01:00
Clinton Gormley 53d0a0a4b9 REST test: Fixed expressions test
Perl reads the number as a string, so instead mapped the field as a long
specifically.

Also, the value returned in JSON is an integer, not a float
2015-02-28 17:02:29 +01:00
Clinton Gormley 4d78026ef1 Updated copyright years to include 2015 2015-02-28 03:18:45 +01:00
Ryan Ernst dd0084de83 Mappings: Lock down _ttl field
While the parser allowed changing field type settings, these would never
have been serialized.  So this change simply removes parsing using
parseField. Backcompat will still work if a user uploads old settings
(they just would never have worked anyways, so we continue ignoring
them with 1.x, and 2.x will now error).

see #8143
closes #9914
2015-02-27 11:11:04 -08:00
Boaz Leskes cc2b00c443 Test: cluster.health rest test should use 0 replicas
Some client test harnesses do not limit the number of replicas based on the number of nodes.
2015-02-25 20:35:39 +01:00
Boaz Leskes 3e32dd985a Recovery: RecoveryState clean up
To support the `_recovery` API, the recovery process keeps track of current progress in a class called RecoveryState. This class currently have some issues, mostly around concurrency (see #6644 ). This PR cleans it up as well as other issues around it:

- Make the Index subsection API cleaner:
- remove redundant information - all calculation is done based on the underlying file map
- clearer definition of what is what: total files, vs reused files (local files that match the source) vs recovered files (copied over). % based progress is reported based on recovered files only.
- cleaned up json response to match other API (sadly this breaks the structure). We now properly report human values for dates and other units.
- Add more robust unit testing
- Detail flag was passed along as state (it's now a ToXContent param)
- State lookup during reporting is now always done via the IndexShard , no more fall backs to many other classes.
- Cleanup APIs around time and move the little computations to the state class as opposed to doing them out of the API

I also improved error messages out of the REST testing infra for things I run into.

Closes #6644
Closes #9811
2015-02-25 17:34:22 +01:00
Boaz Leskes 06417a85e0 Test: cat health api didn't disable headers 2015-02-25 16:17:47 +01:00
Boaz Leskes 6953777c3a API: add pending tasks count to cluster health
The number of current pending tasks is useful to detect and overloaded master. This commit adds it to the cluster health API. The complete list can be retrieved from the dedicated pending tasks API.

It also adds rest tests for the cluster health variants.

Closes #9877
2015-02-25 14:58:44 +01:00
Alexander Reelsen 5de564a19f Testing: Remove plus sign from YAML test due to encoding issue
The plus sign is not treated correctly in encoding and can lead
to problems, if the search request is encoded as HTTP parameter
instead of the HTTP body.

Relates #9769
2015-02-19 18:35:39 +01:00
Alexander Reelsen 2dc70597c2 Testing: Added REST tests to check if expression scripting works 2015-02-19 09:15:28 +01:00
Boaz Leskes c9480783b3 Rest: expose `master_timeout` flag on `GET _template` & `HEAD _template`
These are master level operation but we currently don't expose this parameter

Closes #9688
2015-02-13 11:25:26 +01:00
Clinton Gormley 75e7ce9c51 REST Test: Lucene segment format can have two or three numbers: 4.10.2 or 4.9 2015-02-12 19:59:43 +01:00
Clinton Gormley 93fe4495ae REST tests - add feature groovy_scripting to skip tests if groovy not enabled 2015-02-12 18:08:35 +01:00
Clinton Gormley 36758ce81b Fixed cat.segments and indices.segments tests - bad YAML indenting 2015-02-12 17:35:19 +01:00
Christoph Büscher 41befaf6b5 Change behaviour of indices segments api to allow no indices
Using '_cat/segments' or the indices segments api without matching any index
now returns empty result instead of throwing IndexMissingException.

Closes #9219
2015-02-12 14:42:15 +01:00
javanna 4e94be8a37 [TEST] Introduce basic validation of our REST spec
Whenever we have an api that supports GET with a body, we always support the POST method too, as well as providing the body as a query_string parameter called `source`. Our REST spec should reflect this convention. FIxed them and introduced a hard check at parse time in our Java REST tests runner, which will cause the tests to fail if spec are not compliant.

Closes #9629
2015-02-12 22:25:17 +11:00
javanna 9c847db8af Percolate api: support encoded body as query string param consistently
The percolate api doesn't parse the encoded body provided as `source` query string parameter, when percolating an existing document. Fixed and added REST test that would have caught this since we randomly use GET + encoded `source` param instead of GET + request body in our java runner (the perl runner does the same too).

Closes #9628
2015-02-11 08:53:04 +11:00
Ryan Ernst c6968883a7 Mappings: Remove support for new indexes using path setting in
object/nested fields or index_name in any field

Backcompat is still here for indexes created before 2.0.

closes #6677
2015-02-05 12:44:43 -08:00
Simon Willnauer 0c5599e1d1 [ENGINE] Remove full flush / FlushType.NEW_WRITER
The `full` option and `FlushType.NEW_WRITER` only exists to allow
realtime changes to two settings (`index.codec` and `index.concurrency`).
Those settings are very expert and don't really need to be updateable
in realtime.
2015-02-04 17:38:05 +01:00
Clinton Gormley 988f35a7da REST: Add skip "stash_in_path" feature to nodes.info/20_transport test
Required for clients that don't yet do stash lookups when resolving
fieldnames like "nodes.$master.transport"
2015-02-02 17:11:04 +01:00
Clinton Gormley e7ac5f296e REST-spec: Allow stashed values to be used in property names as well
Fix the nodes.info/20_transport test to use the master node, rather
than to rely on applying a regex to the whole $body
2015-02-02 16:12:41 +01:00
Alexander Reelsen 59f8c0951a Netty Transport: Add profiles to transport infos
Until now, there was no possibility to expose infos about configured
transport profiles. This commit adds the ability to expose those
information in the TransportInfo class.

The channel was well as the netty pipeline handler now also contain
the profile they were configured for, as this information cannot be
extracted elsewhere.

In addition, each profile now can set its own publish host and port,
which might be needed in case of portforwarding or using docker.

Closes #9134
2015-02-02 08:17:55 +01:00
Andrew Ochsner f95fa83e5b Closes #9398 2015-01-23 18:06:12 +01:00
Clinton Gormley 6aff2b9f65 Changed all rest-spec definitions of expand_wildcards to be an enum
with values: open, closed, none, all

Relates to #9295
2015-01-23 16:05:59 +01:00
Christoph Büscher eeb96db76b Rest: Adding support of multi-index query parameters for _cluster/state
Adding missing support for the multi-index query parameters 'ignore_unavailable',
'allow_no_indices' and 'expand_wildcards' to '_cluster/state' API. These
parameters are supposed to be supported for APIs that work across multiple indices.
So far overwriting the default settings per REST call was not possible which is
fixed here.

Closes #5229
Closes #9295
2015-01-22 16:47:14 +01:00
David Pilato 052645903a Rest: remove status code from main action
Today we give the HTTP status back within the HTTP response itself and within the JSON response as well:

```sh
curl localhost:9200/
```

```js
{
  "status" : 200,
  "name" : "Red Wolf",
  "version" : {
    "number" : "2.0.0",
    "build_hash" : "6837a61d8a646a2ac7dc8da1ab3c4ab85d60882d",
    "build_timestamp" : "2014-08-19T13:55:56Z",
    "build_snapshot" : true,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}
```
2015-01-12 12:37:46 +01:00
Martijn van Groningen a88dd36df4 Test: Only run the test that verifies the `pending` field in the `_shards` header if the test cluster has two are more data nodes. 2015-01-12 10:29:23 +01:00
Christoph Büscher 04cb09f44c [TEST] Add missing docs and tests for '_cat/segments'
The '_cat/segments' api was missing docs and a rest test which are added here.

Closes #5856
2015-01-09 12:29:11 +01:00
Martijn van Groningen ca4f27f40e Core: Added `_shards` header to all write responses.
The header indicates to how many shard copies (primary and replicas shards) a write was supposed to go to, to how many
shard copies to write succeeded and potentially captures shard failures if writing into a replica shard fails.

For async writes it also includes the number of shards a write is still pending.

Closes #7994
2015-01-08 18:10:08 +01:00
Colin Goodheart-Smithe ecfe72ebcc Indices API: Fix to make GET Index API consistent with docs
This fix ensures that calls to the GET alias/mappings/settings/warmers APIs return the aliases/mappings/settings/warmers object even if there is no content within them.. This make them consistent with the GET Index API docs and the breaking changes in 1.4 docs

Closes #9148
2015-01-08 08:48:44 +00:00
Martijn van Groningen 20f7be378b Removed parent parameter from update request, because it is just sets the routing.
The routing option should be used instead. The parent a child document points to can't be updated.

Closes #4538
2015-01-07 10:26:20 +01:00
Michael McCandless 242e631e95 Core: ignore known idle threads by default in /_nodes/hot_threads
Add a new ignore_idle_threads boolean option (default true) to
/_nodes/hot_threads, to filter out threads in known idle places like
waiting on a socket select or on pulling the next task from an empty
queue.

Closes #8985

Closes #8908
2014-12-17 11:59:31 -05:00
Alex Ksikes 86e1655e4b Term Vectors: support for version and version_type
This commit adds support for version and version_type to the Term Vectors API.
This could be useful in the following case whereby the user gets a document
and later wants to generate its TVs. With version, this would ensure that only
the TVs of that particular document are generated, and error out if the
document has been updated in between.

Closes #7480
2014-12-17 15:43:15 +01:00
Martijn Laarman bc76032fdd Documented the new terminate_after querystring option on search as implemented in #6885 2014-12-17 14:49:05 +01:00
Lee Hinman 9b18c44b67 Default _cat APIs to verbose
`?v=false` can be used if the headers are not desired.

Resolves #8922

Fixes #8927
2014-12-15 12:51:59 +00:00
Martijn Laarman bb6e539bc7 updated (m)termvectors api specs to reflect #7530 which made it into 1.4 2014-12-12 11:04:21 +01:00
Honza Král 750310cf94 [API] indices.upgrade's wait_for_completion param defaults to false 2014-12-11 17:29:14 +01:00
Sebastian Ziebell ab001cf8e3 API endpoint cluster.put_settings accepts timeout & master_timeout query parameters
Closes #8769
2014-12-04 09:14:20 +01:00
Alex Ksikes 256712640f MLT Query: Support for ignore docs
Adds a `ignore_like` parameter to the MLT Query, which simply tells the
algorithm to skip all the terms from the given documents. This could be useful
in order to better guide nearest neighbor search by telling the algorithm to
never explore the space spanned by the given `ignore_like` docs. In essence we
are interested about the characteristic of a given item, but not of the ones
provided by `ignore_like`, thereby forcing the algorithm to go deeper in its
selection of terms. Note that this is different than simply performing a must
not boolean query on the unliked items. The syntax is exactly the same as the
`like` parameter.

Closes #8674
2014-11-28 14:48:43 +01:00
Sebastian Ziebell 4dd228d02f Spec: Fixes URL links to documentation
Closes #8656
2014-11-25 19:23:59 +01:00
Karel Minarik d33f1d4bd8 [SPEC] Removed the utilities for generating REST specs/code
The utilities have been migrated to the elasticsearch/elasticsearch-ruby repository.

See: https://github.com/elasticsearch/elasticsearch-ruby/tree/master/elasticsearch-api/utils

Related: elasticsearch/elasticsearch-ruby@dfac51d

Closes #8606
2014-11-25 15:33:44 +01:00
Michael McCandless 856b294441 Core: let Lucene kick off merges
Today, Elasticsearch has a separate merge thread pool checking once
per second (by default) if any merges are necessary, but this is no
longer necessary since we can and do now tell Lucene's
ConcurrentMergeScheduler never to "hard pause" threads when merges
fall behind, since we do our own index throttling.

This change goes back to letting Lucene launch merges as needed, and
removes these two expert settings:

  index.merge.force_async_merge
  index.merge.async_interval

Now merges kick off immediately instead of waiting up to 1 second
before running.

Closes #8643
2014-11-25 04:13:57 -05:00
Sebastian Ziebell e0d6e0a6d3 Spec: Fixes a few locations of the params array in JSON specification
A few locations of the `params` object in the JSON specification were not located in the `url` block.

Closes #8607
2014-11-24 20:43:33 +01:00
Alex Ksikes 1959275622 Term Vectors: More consistent naming for term vector[s]
We speak of the term vectors of a document, where each field has an associated
stored term vector. Since by default we are requesting all the term vectors of
a document, the HTTP request endpoint should rather be called `_termvectors`
instead of `_termvector`. The usage of `_termvector` is now deprecated, as
well as the transport client call to termVector and prepareTermVector.

Closes #8484
2014-11-21 14:06:44 +01:00
nwarz e77f9720d2 Mapping: Fixes Merging of default analyzer
Fixed behaviour where two representations of the default index analyzer weren't being treated as equivalent. Added REST test to confirm fix.

Closes #2716
2014-11-14 09:58:26 +00:00
Clinton Gormley f9810e591e Rest API: indices.delete should have "index" as a required param 2014-11-10 17:06:43 +01:00
Chris Mattmann f47fb6b1cf fix mention of Apache License, version 2.
Closes #8400
2014-11-08 19:14:51 +01:00
Veres Lajos 4059e4ac86 typo fixes - https://github.com/vlajos/misspell_fixer
Closes #8323
2014-11-08 18:55:57 +01:00
javanna 3afe86d447 [TEST] fix term_vector realtime test to call wait for green and use no replicas
If a shard (e.g. replica) gets initialized after we indexed the document it gets refreshed internally and we find the doc and its term_vectors, thus the test fails
2014-11-07 20:05:14 +01:00
Adrien Grand 9ea25df649 Switch to murmurhash3 to route documents to shards.
We currently use the djb2 hash function in order to compute the shard a
document should go to. Unfortunately this hash function is not very
sophisticated and you can sometimes hit adversarial cases, such as numeric ids
on 33 shards.

Murmur3 generates hashes with a better distribution, which should avoid the
adversarial cases.

Here are some examples of how 100000 incremental ids are distributed to shards
using either djb2 or murmur3.

5 shards:
Murmur3: [19933, 19964, 19940, 20030, 20133]
DJB:     [20000, 20000, 20000, 20000, 20000]

3 shards:
Murmur3: [33185, 33347, 33468]
DJB:     [30100, 30000, 39900]

33 shards:
Murmur3: [2999, 3096, 2930, 2986, 3070, 3093, 3023, 3052, 3112, 2940, 3036, 2985, 3031, 3048, 3127, 2961, 2901, 3105, 3041, 3130, 3013, 3035, 3031, 3019, 3008, 3022, 3111, 3086, 3016, 2996, 3075, 2945, 2977]
DJB:     [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 900, 900, 900, 900, 1000, 1000, 10000, 10000, 10000, 10000, 9100, 9100, 9100, 9100, 9000, 9000, 0, 0, 0, 0, 0, 0]

Even if djb2 looks ideal in some cases (5 shards), the fact that the
distribution of its hashes has some patterns can raise issues with some shard
counts (eg. 3, or even worse 33).

Some tests have been modified because they relied on implementation details of
the routing hash function.

Close #7954
2014-11-04 16:32:42 +01:00
Martijn Laarman 006acfe8bf Added missing percolate API parameters (percolate_routing, percolate_preference) to the REST API Spec
Closes #7173
2014-10-28 19:34:34 +01:00
Clinton Gormley b94b3b2bcd Tests: Refixed the put_alias tests
The missing/null index parameter is in the URL, not the body.
So it should be required, and throw a param error if not provided.

Relates #7863
2014-10-28 13:39:05 +01:00
Clinton Gormley f9b5906871 Test: Change missing/null alias test to catch a request error, not a param error
Relates: #7863
2014-10-28 11:43:10 +01:00
Spencer 055e766be7 Updated paths to be inline with #8240 2014-10-27 23:08:19 -07:00
tlrx 8c864cf3f6 Cat Recovery API: Reverting changes introduced with commit e1c75bae87
Adding these 2 headers to the CAT Recovery made the CI tests hanging for a loooong time.

Related to #8041
2014-10-27 20:49:58 +01:00
Zachary Tong f5b2dfd052 Aliases: Throw exception if index is null or missing when creating an alias
Fixes a bug where alias creation would allow `null` for index name, which thereby
applied the alias to _all_ indices.  This patch makes the validator throw an
exception if the index is null.

```bash
POST /_aliases
{
   "actions": [
      {
         "add": {
            "alias": "empty-alias",
            "index": null
         }
      }
   ]
}
```
```json
{
   "error": "ActionRequestValidationException[Validation Failed: 1: Alias action [add]: [index] may not be null;]",
   "status": 400
}
```

The reason this bug wasn't caught by the existing tests is because
the old test for nullness only validated against a cluster which had
zero indices.  The null index is translated into "_all", and since
there are no indices, this fails because the index doesn't exist.
 So the test passes.

However, as soon as you add an index, "_all" resolves and you get the
situation described in the original bug report:  null index is
accepted by the alias, resolves to "_all" and gets applied to everything.

The REST tests, otoh, explicitly tested this bug as a real feature and therefore
passed.  The REST tests were modified to change this behavior.

Fixes #7863
2014-10-27 14:39:01 -04:00
tlrx 96e62b3c1b [TESTS Fix wrong assertion in test introduced by #8128 2014-10-27 11:11:54 +01:00
tlrx e1c75bae87 Cat API: Add node name to _cat/recovery
Add source_node and target_node fields to the recovery cat API. Also fixed and updated the documentation which was not complete concerning fields names.

Closes #8041
2014-10-27 09:47:26 +01:00
Alex Ksikes c13f5f21de Term Vectors: support for distributed frequencies
Adds distributed frequencies support for the Term Vectors API. A new parameter
called `dfs` is introduced which defaults to `false`.

Closes #8144
2014-10-23 13:59:59 +02:00
Adrien Grand f4ee3f25e4 Mappings: Store _timestamp by default.
Storing `_timestamp` by default means that under the default configuration, you
would have all the information you need in order to reindex into a different
index.

Close #8139
2014-10-20 12:17:26 +02:00
Spencer 249a145a5c Update cat.nodes regex
Support for multi-word node names like 'Henry "Hank" McCoy'
2014-10-15 14:33:41 -07:00
Chris Earle 04926954e2 Updating test to support Windows file descriptor count (-1) 2014-10-15 14:02:32 -05:00
Chris Earle 29c5aaa1d3 Fixing test to allow decimal numbers for load. 2014-10-15 11:54:12 -05:00
Chris Earle 2d8a140ed8 Add file descriptor details to cat/nodes
cat/nodes currently does not report any details related to file descriptors. This adds the current number in use, the maximum number available as well as their ratio (percentage) to cat/nodes as hidden-by-default metrics. In addition, this also adds current heap usage (as a non-percentage of ts max) and ram usage (as a non-percerntage of its max) to allow tools to provide more granularity.

Closes #7652
2014-10-15 10:18:41 -05:00
Clinton Gormley dfcc0f97f0 Spec: Removed flush and max_num_segments from indices.upgrade 2014-10-11 17:19:06 +02:00
Clinton Gormley 1e47f02891 Rest: Added missing parameters to indices.upgrade 2014-10-11 16:40:45 +02:00
Clinton Gormley 20a901964c Spec: Added human flag to indices.get_upgrade 2014-10-10 17:26:36 +02:00
Karel Minarik b1d4cec7ab [SPEC] Separated the "Upgrade Index" API into two methods
* `get_upgrade` => `GET _upgrade`  -- Return the status
* `upgrade`     => `POST _upgrade` -- Perform the operation

Original specification part of c021f22523.

Related: #7884, #7922
2014-10-09 16:19:58 +02:00
Ryan Ernst c06c10bbb0 Remove deprecations from master (follow up to #7922) 2014-10-07 08:35:11 -07:00
Ryan Ernst c021f22523 Add Upgrade API
This commit does the following:
* Add the new API at the rest layer, being backed by the optimize API
  with upgrade flag, and segments api to find upgrade status.
* Add `upgrade` flag to optimize API, and deprecate `force` flag (will
  remove in master)
* Add test for both synchronous and async upgrade

closes #7884
closes #7922
2014-10-07 08:09:50 -07:00
Igor Motov 555bfcb02b [SNAPSHOT] Add repository validation
Fixes #7096
2014-10-07 10:50:16 -04:00
David Pilato f0052a58d6 Admin: show open and closed indices in _cat/indices
When asking for `GET /_cat/indices?v`, you can now retrieve closed indices in addition to opened ones.

```
health status index              pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .marvel-2014.05.21   1   1       8792            0     21.7mb         21.7mb
       close  test
yellow open   .marvel-2014.05.22   1   1       3871            0     10.7mb         10.7mb
red    open   .marvel-2014.05.27   1   1
```

Closes #7907.
Closes #7936.
2014-10-03 13:16:14 +02:00
Alex Ksikes c4830cf862 Term Vectors: support for realtime
By default term vectors are now realtime, as opposed to previously near
realtime. If they are not found in the index, they will be generated on the
fly. The document is fetched from the transaction log and treated as an
artificial document. One can set `realtime` parameter to `false` in order to
disable this functionality. This consequently makes the MLT query realtime in
fetching documents, as it previsouly used to be before switching from using
the multi get API to the mtv API.

Closes #7846
2014-10-03 09:26:47 +02:00
Honza Král 3aa4ac9caa [TEST] Make mlt rest tests pass on a single node cluster 2014-09-30 22:42:51 +02:00
Alex Ksikes b118558962 MLT Query: Support for artificial documents
Previously, the only way to specify a document not present in the index was to
use `like_text`. This would usually lead to complex queries made of multiple
MLT queries per document field. This commit adds the ability to the MLT query
to directly specify documents not present in the index (artificial documents).
The syntax is similar to the Percolator API or to the Multi Term Vector API.

Closes #7725
2014-09-29 15:49:13 +02:00
Spencer 19f5a86c21 Update indices.get_mapping.json 2014-09-25 09:53:27 -07:00
javanna 07ca08dbed [TEST] improved regular scroll REST test
Added sort clause by field and checked docs returned each time
2014-09-25 13:02:16 +02:00
javanna dc1ef7e670 [TEST] improved regular scroll REST test
The intermediate document that gets indexed has now a non existing id, and we make sure it doesn't get returned.
2014-09-25 11:46:05 +02:00
javanna f52375198b [TEST] add regular scroll REST test
Closes #7860
2014-09-25 11:03:12 +02:00
Brian Murphy 8e742c2096 Indexed Scripts/Templates : Cleanup
This contains several cleanups to the indexed scripts.
Remove the unused FetchSourceContext from the Get request..
Add lang,_version,_id to the REST GET API.
Removes the routing from GetIndexedScriptRequest since the script index is a single shard that is replicated across all nodes.
Fix backward compatible template file reference
Before 1.3.0 on disk scripts could be referenced by requesting
````
_search/template

{
  "template" : "ondiskscript"
}
````
This was broken in 1.3.0 by requiring
````
{
  "template" :
  {
    "file" : "ondiskscript"
  }
}
````
This commit restores the previous behavior.
Remove support for preference, realtime and refresh
These parameters don't make sense anymore for indexed scripts as we always force the preference to _local and
always refresh after a Put to the indexed scripts index.

Closes #7568
Closes #7559
Closes #7647
Closes #7567
2014-09-19 11:59:08 +01:00
javanna 7e0481d906 More Like This API: remove unused search_query_hint parameter
Closes #7691
2014-09-11 17:34:54 +02:00
Clinton Gormley 269b91c688 Spec: Fixed the docs URL for indices.get and indices.exists 2014-09-11 16:39:46 +02:00
Honza Král 480b90cfd6 [API] Fix minor issues with indices.get definition and tests
mark index param as required
make body match json, not string containing json
2014-09-11 14:36:11 +02:00
Colin Goodheart-Smithe 5fe782b784 Indices API: Added GET Index API
Returns information about settings, aliases, warmers, and mappings. Basically returns the IndexMetadata. This new endpoint replaces the /{index}/_alias|_aliases|_mapping|_mappings|_settings|_warmer|_warmers and /_alias|_aliases|_mapping|_mappings|_settings|_warmer|_warmers endpoints whilst maintaining the same response formats.  The only exception to this is on the /_alias|_aliases|_warmer|_warmers endpoint which will now return a section for 'aliases' or 'warmers' even if no aliases or warmers exist. This backwards compatibility change is documented in the reference docs.

Closes #4069
2014-09-11 11:19:21 +01:00
Boaz Leskes 4f8ddd97bf [Rest] reroute API response didn't filter metadata
By default the reroute API should return the new cluster state, excluding the metadata. It was however it was wrongly using an old parameter (filter_metadata) and thus failed to do so. This commits restores but wiring it to the correct `metric` parameter. We also add an enum representing the possible metrics, to avoid similar future mistakes.

Closes #7520
Closes #7523
2014-09-10 14:48:06 +02:00
Martijn van Groningen 52f1ab6e16 Core: Added the `index.query.parse.allow_unmapped_fields` setting to fail queries if they refer to unmapped fields.
The percolator and filters in aliases by default enforce strict query parsing.

Closes #7335
2014-09-09 15:00:47 +02:00
javanna a857798e1c Indexed scripts: make sure headers are handed over to internal requests and streamline versioning support
The get, put and delete indexed script apis map to get, index and delete api and internally create those corresponding requests. We need to make sure that the original headers are handed over to the new request by passing the original request in the constructor when creating the new one.

Also streamlined the support for version and version_type in the REST layer since the parameters were not consistently parsed and set to the internal java API requests.

Modified the REST delete template and delete script actions to make use of a client instead of using the `ScriptService` directly.

Closes #7569
2014-09-04 16:00:32 +02:00
Boaz Leskes 0e6bb1f28b [Rest] Add the cluster name to the "/" endpoint
The root endpoint returns basic information about this node, like it's name and ES version etc. The cluster name is an important information that belongs in that list.

Closes #7524
2014-09-01 10:05:11 +02:00
Simon Willnauer 91b8498cec [TEST] Port can have more or less than 4 digits 2014-08-29 08:57:35 +02:00
Clinton Gormley ce0b2ade9c Test: Fixed the regex in cat.indices/10_basic.yaml
And renamed the file from .yml to .yaml
2014-08-25 16:13:16 +02:00
Simon Willnauer 6950c38a04 Tests: Improve test coverage.
Close #7428
2014-08-25 11:56:38 +02:00
Alex Ksikes 62ef4a30dc Term vector API: return 'found: false' for docs between index and refresh
Closes #7121
2014-08-21 09:58:49 +02:00
Colin Goodheart-Smithe fb651f7755 [TEST] fix for wildcard_expansion REST tests
removed timeout from wait_for_green and set number of replicas to 0 so client tests can pass on single node.
2014-08-18 14:36:43 +01:00
Colin Goodheart-Smithe 925a4ba28b [TEST] fix to wildcard_expansion tests to wait for green status 2014-08-18 10:18:12 +01:00
Colin Goodheart-Smithe f4d75f0212 REST API: Allows all options for expand_wildcards parameter
This change means that the default settings for expand_wildcards are only applied if the expand_wildcards parameter is not specified rather than being set upfront. It also adds the none and all options to the parameter to allow the user to specify no expansion and expansion to all indexes (equivalent to 'open,closed')

Closes #7258
2014-08-15 12:50:11 +01:00
javanna 6d3bcc4451 Java API: add index, type and id to ExplainResponse
Index, type and id were returned as part of the REST explain api response, but not through java api. That info was read out of the request, relying on the fact that the index would get overridden with the concrete one within that same request.

Closes #7201
2014-08-08 12:52:03 +02:00
Clinton Gormley 9d65db4dba Test: Trimmed trailing whitespace to make valid YAML 2014-08-06 15:05:46 +02:00
Clinton Gormley 11f8edd74a REST spec: Added missing query_cache param to clear_cache, nodes.stats and indices.stats
Relates to #7167 and #7161
2014-08-06 13:32:33 +02:00
Shay Banon e6e2781ee7 [Query Cache] Add a request level flag to control query cache
A request level flag, defaults to be unset, to control the query cache. When not set, it defaults to the index level settings, when explicitly set, will override the index level setting
closes #7167
2014-08-05 18:28:49 +02:00
markharwood e6b459cb9f Update API enhancement - add support for scripted upserts.
In the case of inserts the UpdateHelper class will now allow the script used to apply updates to run on the upsert doc provided by clients. This allows the logic for managing the internal state of the data item to be managed by the script and is not reliant on clients performing the initialisation of data structures managed by the script.

Closes #7143
2014-08-05 16:52:44 +01:00
Areek Zillur 1d581e6286 Search Exists API: Checks if any matching documents exist for a given query
Implements a new Exists API allowing users to do fast exists check on any matched documents for a given query.
This API should be faster then using the Count API as it will:
 - early terminate the search execution once any document is found to exist
 - return the response as soon as the first shard reports matched documents

closes #6995
2014-07-31 15:42:30 -04:00
Spencer Alger a848b658e8 [TEST] Rest API: cat.count regexp matchers now explicitely test for newline 2014-07-30 10:19:24 -07:00
Simon Willnauer bd51d7a07f Add `wait_if_ongoing` option to _flush requests
This commit adds the ability to force blocking on the flush operaition
to make sure all files have been written and synced to disk. Without
this option a flush might be executing at the same time causing the
current flush to fail and return before all files being synced.

Closes #6996
2014-07-24 15:34:53 +02:00
Lee Hinman 6e25a6a7aa [DOCS] clarify /_cat/fielddata REST api documentation 2014-07-23 16:18:37 +02:00
Clinton Gormley 7bf9d7271a Test: Set number_of_replicas to zero so that wait_for_green succeeds in search.aggregations 2014-07-21 22:41:15 +02:00
Colin Goodheart-Smithe 7223e9959d [TEST] Fixed intermittent failure due to lack of mapping 2014-07-17 09:05:58 +01:00
Clinton Gormley cccc8867b9 REST: Fixed indexed script/template tests
The clients don't have access to the error message
for matching purposes.  Rewritten tests to work
with the clients
2014-07-16 17:31:31 +02:00
Clinton Gormley 65b85566a0 REST: Renamed indexed_script and indexed_template specs
The file name of the REST specs should be the same as
the endpoint which it documents.
2014-07-16 17:31:26 +02:00
Colin Goodheart-Smithe 8260138e59 Aggregations: Fixed Histogram key_as_string bug
The key as string field in the response for the histogram aggregation will now only show if format is specified on the request.

Closes #6655
2014-07-16 11:36:48 +01:00
Brian Murphy cbd2a97abd [DOCS] : Indexed scripts/templates
These are the docs for the indexed scripts/templates feature.
Also moved the namespace for the REST endpoints.

Closes #6851
2014-07-16 10:49:02 +01:00
Brian Murphy f68a06ff99 [TEST] Fix test relying on order of json for matching 2014-07-14 17:06:23 +01:00
Brian Murphy e79b7086de Indexed Scripts/Templates: Add support for storing/deleting/getting scripts/template from an index.
This change allow elasticsearch users to store scripts and templates in an index for use at search time.
Scripts/Templates are stored in the .scripts index. The type of the events is set to the script language.
Templates use the mustache language so their type is be "mustache".
Adds the concept of a script type to calls to the ScriptService types are INDEXED,INLINE,FILE.
If a script type of INDEXED is supplied the script will be attempted to be loaded from the indexed, FILE will
look in the file cache and INLINE will treat the supplied script argument as the literal script.
REST endpoints are provided to do CRUD operations as is a java client library.
All query dsl points have been upgraded to allow passing in of explicit script ids and script file names.
Backwards compatible behavior has been preserved so this shouldn't break any existing querys that expect to
pass in a filename as the script/template name. The ScriptService will check the disk cache before parsing the
script.

Closes #5921 #5637 #5484
2014-07-14 14:37:55 +01:00
Spencer Alger c268e2d0ef [API SPEC] Update indicies.stats group param 2014-07-08 11:25:13 -07:00
javanna 8dedbd01df Core: don't replace indices within the ActionRequest with the concrete ones, and make sure check blocks is executed on concrete indices
Concrete indices is now called multiple times when needed instead of changing what's inside the incoming request with the concrete indices. Ideally we want to keep the original aliases or indices or wildcard expressions in the request.

Also made sure that the check blocks is done against the concrete indices, which wasn't the case for delete index, delete mapping, open index, close index, types exists and indices exists.

Closes #6694
Closes #6777
2014-07-08 14:18:26 +02:00
Boaz Leskes 83770c2583 [Test] Revert temporary fixes for update mapping on recovery issue
This reverts:
"Test: Temporarily change delete/put_mapping to wait for green": commit e408f8f638c2dd97a3ec86c8a9ac940f43ab37a0.
"[TEST] wait for green to update mapping": commit b3641a2ee6eb23318d49f5f04b39149e70c2b65b.
2014-07-07 22:27:25 +02:00
Clinton Gormley 34893c0570 Test: Temporarily change delete/put_mapping to wait for green
Delete and update mapping execution order has changed with #6762
2014-07-07 19:02:36 +02:00
Adrien Grand a96f9a7c83 Templates: GET templates doesn't honor the `flat_settings` parameter.
Close #6671
2014-07-02 08:42:31 +02:00
Lee Hinman 5c6d28240f Switch to Groovy as the default scripting language
This is a breaking change to move from MVEL -> Groovy
2014-06-25 12:15:12 +02:00
Britta Weber 1cbeaf6c45 significant terms: fix json response
Commit fbd7c9aa5d introduced a regression that caused
the min_doc_count to be equal to the number of documents in the
background set. As a result no buckets were built when the
response for significant terms was created.
This only affected the final XContent response.

closes #6535
2014-06-18 18:51:34 +02:00
Zachary Tong ed62f90620 [TEST] Remove indentation on indices.stats test setups
The new indices.stats tests have the setup section indented, which
can cause problems for some yaml parsers.
2014-06-18 11:59:40 -04:00
Clinton Gormley eabd4abf57 Rest-Spec: search was missing the track_scores param 2014-06-12 18:25:20 +02:00
Honza Král 681e9fa522 [TEST] remove accidental double escape in yaml tests 2014-06-12 01:03:36 +02:00
Clinton Gormley 0859f2e588 Tests: Java test runner can't handle array responses 2014-06-10 20:33:04 +02:00
Clinton Gormley eb3e0fb931 Tests: Fixed indices.stats types test when run with replicas 2014-06-10 19:10:18 +02:00
Clinton Gormley bb15def36e Stats: Bugfixes and enhancements to indices stats API
Bugs:
* "groups" and "types" were being ignored
* "completion_fields" as wildcards were not being resolved to fieldnames

Enhancements:
* Made "groups" and "types" support wildcards
* Added missing tests

Closes #6390
2014-06-10 17:35:49 +02:00
javanna 11f7c31852 Put index template api: unified PUT/POST behaviour in relation to create parameter
The put index template api supports the create parameter (defaults to false), which tells whether the template can replace an existing one with same name or not. Unified its behaviour between PUT and POST method, whereas the POST would previously force create to true.

Added create parameter to the rest spec (was missing before) and a REST test for create true scenario.
2014-06-06 15:45:05 +02:00
javanna 4a2410135f [TEST] added wait for relocationg shards to cat.shards REST test
Adding the second index might cause relocation of existing shards depending on the number of nodes available, let's wait for relocation to be finished before going ahead and checking the cat shards outpu
2014-06-03 10:48:11 +02:00
Clinton Gormley bec51a618f Test: Python doesn't support atomic groups in regexes. Removing from cat.recovery 2014-05-29 16:57:09 +02:00
Karel Minarik 0e920c17dd [SPEC] Added the `percolator_format` URL parameter to the "Percolate" API spec 2014-05-23 18:26:17 +02:00
Karel Minarik 81cddacffa [SPEC] Removed the URL parameter `indices_boost` from the "Search" API
Related: elasticsearch/elasticsearch-ruby#29
2014-05-22 15:29:46 +02:00
javanna de9fca9cb2 [TEST] indices.open REST tests need 0 replicas
With replicas the search requests might go to shards that are not ready, since we have a wait for yellow. We can now switch to wait for green as well.
2014-05-22 11:04:28 +02:00
javanna 1b996defcf [TEST] cat.allocation/One index: allow leading spaces in unassigned shards lines 2014-05-22 10:10:50 +02:00
Boaz Leskes 9f10547f4b Allow 0 as a valid external version
Until now all version types have officially required the version to be a positive long number. Despite of this has being documented, ES versions <=1.0 did not enforce it when using the `external` version type. As a result people have succesfully indexed documents with 0 as a version. In 1.1. we introduced validation checks on incoming version values and causing indexing request to fail if the version was set to 0. While this is strictly speaking OK, we effectively have a situation where data already indexed does not match the version invariant.

To be lenient and adhere to spirit of our data backward compatibility policy, we have decided to allow 0 as a valid external version type. This is somewhat complicated as 0 is also the internal value of `MATCH_ANY`, which indicates requests should succeed regardles off the current doc version. To keep things simple, this commit changes the internal value of `MATCH_ANY` to `-3` for all version types.

Since we're doing this in a minor release (and because versions are stored in the transaction log), the default `internal` version type still accepts 0 as a `MATCH_ANY` value. This is not a problem for other version types as `MATCH_ANY` doesn't make sense in that context.

Closes #5662
2014-05-16 22:10:16 +02:00
Clinton Gormley a7362d9ac7 [TEST] Changed the possessive matching syntax in cat.recovery
regex to use an independent subexpression. Possessive
modifiers not supported on older Perl
2014-05-13 11:02:36 +02:00
Clinton Gormley 29ab31b351 [TEST] cat.recovery regex still causing occasional JVM regex stackoverflows.
Try making the capture possessive to prevent recursion
2014-05-12 15:47:51 +02:00
javanna 3d63bac51d Fixed validate query parsing issues
Made sure that a match_all query is used when no query is specified and ensure no NPE is thrown either.
Also used the same code path as the search api to ensure that alias filters are taken into account, same for type filters.

Closes #6111 Closes #6112 Closes #6116
2014-05-12 12:49:25 +02:00
Martijn van Groningen 145efbf6ea Return missing (404) is a scroll_id is cleared that no longer exists.
Closes #5730
2014-05-12 09:43:56 +02:00
Clinton Gormley 4fefe628b0 [TEST] Aborting a non-existent benchmark throws a missing request 2014-05-10 13:09:30 +02:00
Lee Hinman 90e019b4a4 [TEST] Add test specifying fields field /_cat/fielddata REST test 2014-05-09 23:57:23 +02:00
javanna 6678da8c28 [TEST] randomly added node.bench=true to client node in test cluster and re-enabled REST benchmark tests based on number of bench nodes available
In our REST tests we already have support for features and skip sections that allow to skip tests if a feature is not supported.
We can then add a skip section based on the benchmark feature to the benchmark tests and execute them only when they are supported, knowing that they need at least a node with node.bench settings within the cluster. We can check that this requirement is met by calling the nodes info api.

This way we can dynamically decide whether to execute those tests or not and we don't need to have a node.bench around all the time. In fact, given that the REST tests use the GLOBAL cluster, we want to be able to randomize settings as much as possible and run tests against default settings as well. Also, this mechanism can be easily supported by the external cluster implementation that is used during the release process.

Introduced ability to disable benchmark nodes which is needed by BenchmarkNegativeTest.
2014-05-09 23:36:00 +02:00
Lee Hinman 7847ec0a6c Fix typo in path specification for /_cat/fielddata 2014-05-09 15:03:51 +02:00
Lee Hinman e7e4ef859a Add /_cat/fielddata to display fielddata usage
Closes #4593
2014-05-09 13:18:02 +02:00
Clinton Gormley a972aaa7ae [TEST] Temporarily added a wait-for-yellow to delete_mapping tests
Should be removed once #5997 is fixed
2014-05-09 12:04:34 +02:00
Clinton Gormley e0a45459a4 [TEST] cat.recovery with replicas can have 100.0% 2014-05-08 15:40:28 +02:00
Clinton Gormley eb6d3e3eb3 [TEST] Fixed pathological regex in cat.recovery
Fixes #6070
2014-05-08 14:15:47 +02:00
Andrew Selden f739abe89e Fix stack overflow in _cat/recovery test.
This fixes a stack overflow in the test for the _cat/recovery API.
The regular expression that tests the response body was modified to
handle large responses properly.
2014-05-07 13:58:00 -07:00
Andrew Selden 35f046e24f Fix _cat/allocation rest test
The rest test for _cat/allocation was failing due to a regular
expression not accounting for space-padded right-justified text.

Also added Improvements to regular expressions to be smarter about optional values
and to use '+' instead of '*' where applicable.
2014-05-07 13:10:31 -07:00
Clinton Gormley 5937cefd85 Added char_filters param to indices.analyze spec 2014-05-05 18:07:50 +02:00
Alexander Reelsen d356881664 [REST] Missing scroll id now returns 404
A bad/non-existing scroll ID used to return a 200, however a 404 might be more useful.
Also, this PR returns the right Exception (SearchContextMissingException) in the Java API.

Additionally: Added StatusToXContent interface and RestStatusToXContentListener listener, so
the appropriate RestStatus can be returned

Closes #5729
2014-05-05 17:37:26 +02:00
Alexander Reelsen d4fcf23057 Cluster State API: Remove index template filtering
The possibility of filtering for index templates in the cluster state API
had been introduced before there was a dedicated index templates API. This
commit removes this support from the cluster state API, as it was not really
clean, requiring you to specify the metadata and the index templates.

Closes #4954
2014-05-05 14:54:14 +02:00
javanna e96e634d10 [TEST] fixed _cat/thread_pool REST tests with local transport, in case the transport port is not available and gets returned as '-'
Re-enabled REST tests suite

Closes #6033
2014-05-04 22:10:03 +02:00
Zachary Tong f4c5cde8af [TEST] Replace folded blocks with literal blocks
The regex tests are formatted with blocks for readability.  Previously,
they were formatted using folded style blocks (e.g. using `>`). Folded
blocks convert newlines into spaces.  This is problematic for our regex,
since comments can only be terminated with a newline.

Effectively, anything after a comment will be commented out, making many
of the regex "silently pass".

This commit replaces them with scalar-style blocks (e.g. using `|`), which
treats newlines as significant, and thus correctly terminates comments
inside the regex.

Also fixes a regex test (`cat.thread_pool/10_basic.yaml`) that started
to fail after the block was fixed.  The test was missing a `\s+` before
the closing newline.
2014-05-02 18:30:48 -04:00
Spencer Alger 9038db7bfc [REST-SPEC] update to update test, to check for es-js error messages 2014-04-29 14:18:20 -07:00
javanna a414e4f2f3 [TEST] randomly introduced a client node within test cluster
The default number of clients nodes is randomized between 0 and 1, applied to all cluster scopes (global, suite and test). Can be changed through the newly added `@ClusterScope#numClientNodes`.

In our tests we currently refer to nodes in a generic way. All the tests that either stop or start nodes rely on the fact that those nodes hold data though. Made that clearer as that becomes more important when introducing other types of nodes within the test cluster. Reflected this by adapting and renaming the following methods in `TestCluster`:

- ensureAtLeastNumNodes to ensureAtLeastNumDataNodes
- ensureAtMostNumNodes to ensureAtMostNumDataNodes
- stopRandomNode to stopRandomDataNode

and the following ones in `ElasticsearchIntegrationTest`:

- allowNodes to allowDataNodes
- dataNodes to numDataNodes.
- @ClusterScope#numNodes to numDataNodes
- @ClusterScope#minNumNodes to minNumDataNodes
- @ClusterScope#maxNumNodes to maxNumDataNodes

Added facilities to be able to deal with data nodes specifically, like for instance retrieve a client to a data node, or retrieve an instance of a class through guice only from data nodes.

Adapted existing tests to successfully run although there's a node client around.

Fixed _cat/allocation REST tests to make disk.total, disk.avail and disk.percent optional as client nodes won't return that info.

Closes #5949
2014-04-28 16:31:36 +02:00
Clinton Gormley 2dfc77a4ed Removed spec and YAML tests for indices.status
Related #4854
2014-04-28 13:00:08 +02:00
Chris Earle 5528370e24 Added type, max, min, queueSize & keepAlive to _cat/thread_pool
Closes #5366
2014-04-28 12:00:27 +02:00
Boaz Leskes 051beb51a3 Version types `EXTERNAL` & `EXTERNAL_GTE` test for version equality in read operation & disallow them in the Update API
Separate version check logic for reads and writes for all version types, which allows different behavior in these cases.
Change `VersionType.EXTERNAL` & `VersionType.EXTERNAL_GTE` to behave the same as `VersionType.INTERNAL` for read operations.
The previous behavior was fit for writes but is useless in reads.

This commit also makes the usage of `EXTERNAL` & `EXTERNAL_GTE` in the update api raise a validation error as it make cause data to
be lost.

Closes #5663 , Closes #5661, Closes #5929
2014-04-25 23:06:12 +02:00
Britta Weber 8076a31ac1 Throw exception if an additional field was placed inside the "query" body
Currently the parser accepts queries like

```
"query" : {
     "any_query": {
         ...
     },
     "any_field_name":...
}
```

The "any_field_name" is silently ignored. However, this also causes the parser
not to move to the next closing bracket which in turn can lead to additional query
paremters being ignored such as "fields", "highlight",...
This was the case in issue #4895

closes issue #4895
2014-04-25 08:57:06 +02:00
Lee Hinman 54612ee2b5 Add REST API spec for /_search_shards endpoint
Also adds REST test for /_search_shards

Closes #5907
2014-04-23 13:29:29 -06:00
Clinton Gormley 640085c45d Disable benchmark tests until nodes set to run with
-Des.node.bench=true
2014-04-22 21:16:46 +02:00
Clinton Gormley eccbd911de [SPEC] Added path without scroll_id to clear_scroll 2014-04-22 19:38:55 +02:00
Clinton Gormley e2b4fbd1b1 [SPEC] The scroll_id is no longer a required qs param
Can be passed in the body instead
2014-04-22 19:30:51 +02:00
Clinton Gormley 2423c8c57a [SPEC] Added missing list_benchmarks test 2014-04-22 19:27:37 +02:00
Clinton Gormley 5d7263b69e [SPEC] Renamed benchmark endpoints
benchmark.submit -> benchmark
benchmark.list   -> list_benchmarks
benchmark.abort  -> abort_benchmark

And fixed the related tests
2014-04-22 19:14:26 +02:00
Andrew Selden 1f7f72135a Bug fix for hung clients on cluster without benchmark nodes
This is a fix for a bug whereby a cluster that has no nodes started with
-Des.node.bench=true will cause clients to hang if they attempt to
submit a benchmark.

Also adds REST tests to validate fix

Closes #5754
2014-04-21 15:08:50 -07:00
Andrew Selden 7ef36d9d52 Separate benchmark API endpoints
Separates benchmark API endpoints into separate files according to API
funtionality. This makes it easier for our tests and clients.

Closes #5787
2014-04-14 17:36:43 -07:00
Martijn van Groningen 45a1b44759 Each search request should use a new InternalSearchResponse instance even in case when all shards return no hits.
The InternalSearchResponse may get modified afterwards, so a new instance required at all times.
2014-04-11 17:44:21 +07:00
Martijn van Groningen 01794bf8ea Ignored clear scroll rest test 2014-04-11 13:47:41 +07:00
Martijn van Groningen b400129597 The clear scroll apis now optionally accepts a scroll_id in it body.
Closes #5726
2014-04-11 10:39:59 +07:00
Simon Willnauer bceac036af Fix _bench endpoint specs to have the body on the right level in the json object 2014-04-09 23:16:52 +02:00
Andrew Selden e2c8ff92ba Benchmark API
Add an API endpoint at /_bench for submitting, listing, and aborting
search benchmarks. This API can be used for timing search requests,
subject to various user-defined settings.

Benchmark results provide summary and detailed statistics on such
values as min, max, and mean time. Values are reported per-node so that
it is easy to spot outliers. Slow requests are also reported.

Long running benchmarks can be viewed with a GET request, or aborted
with a POST request.

Benchmark results are optionally stored in an index for subsequent
analysis.

Closes #5407
2014-04-09 13:06:55 -07:00
Simon Willnauer a9c8624c67 [TEST] MLT Rest test needs a mapping since we randomized number of nodes/shards 2014-04-09 11:45:55 +02:00
Simon Willnauer b4c506b952 [TEST] use no replicas in MLT tests - doc mappers need to be present on all nodes 2014-04-08 10:07:53 +02:00
javanna f283a9f3a5 [TEST] specified number_of_shards 5 to make sure the two docs end up in different shards that can be refreshed separately 2014-04-07 19:01:19 +02:00
javanna 1ec4f8f04b [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly
ElasticsearchRestTests extends now ElasticsearchIntegrationTest and makes use of our ordinary test infrastructure, in particular all randomized aspects now come for free instead of having to maintain a separate (custom) tests runner

We previously parsed only the tests that needed to be run given the version of the cluster the tests are running against. This doesn't happen anymore as it didn't buy much and it would be harder to support as the tests get now parsed before the test cluster gets started. Thus all the tests are now parsed regardless of their skip sections, afterwards the ones that don't need to be run will be skipped through assume directives.

Fixed REST tests that rely on a specific number of shards as this change introduces also random number of shards and replicas (through randomIndexTemplate)

Closes #5654
2014-04-07 17:08:05 +02:00
Alexander Reelsen f0ae43bbea Made template endpoint compatible with search endpoint
Before this the from/size parameters did not work.
Also updated the rest api spec definition file with all the query_string
parameters.

Fixes #5550
2014-04-03 00:01:30 +02:00
Alexander Reelsen e547e113e1 Geo context suggester: Require precision in mapping
The default precision was way too exact and could lead people to
think that geo context suggestions are not working. This patch now
requires you to set the precision in the mapping, as elasticsearch itself
can never tell exactly, what the required precision for the users
suggestions are.

Closes #5621
2014-04-02 23:51:14 +02:00
Alexander Reelsen 0ff30ade69 ContextSuggester: Adding couple of tests to catch more bugs
A bunch of minor fixes have been included here, especially due
to wrongly parsed mappings. Also using assertions resulted in an
NPE because they were disabled in the distribution.

Closes #5525
2014-03-31 16:16:21 +02:00
Clinton Gormley a809bfbcb2 [TEST] Fixed TTL tests for create, index, update
Were failing on faster boxes
2014-03-31 15:13:14 +02:00
javanna 84770c7d33 [TEST] decreased number of replicas to 0 in refresh tests
Some tests disable refresh and verify that documents cannot be found without an explicit refresh. With at least one replica and waiting for yellow though, it can happen that we start indexing docs when the replicas are not ready yet, and as soon as they become ready they get refreshed, causing the newly added docs to be unexpectedly found. Solution is to disable replicas for these specific tests.
2014-03-31 11:28:34 +02:00
Clinton Gormley 73547648ef [TEST] Update TTL test can fail on faster boxes - improve range of valid vals 2014-03-28 17:23:17 +01:00
javanna 34e9f8e83b [TEST] Removed skip regex sections as all the runners implemented the feature 2014-03-28 12:19:46 +01:00
Kevin Wang ceed22fe00 Add suggest stats
closes #4032
2014-03-28 11:13:54 +01:00
Igor Motov 3ffd0a1dfa Remove deprecated gateways
Closes #5422
2014-03-26 18:10:51 -04:00
Simon Willnauer 09e5ac98fa [TEST] Don't use replicas if only primaries are expected or the first shard in the list shoudl be a primary 2014-03-22 18:36:34 +01:00
Clinton Gormley ec8f404ac7 Renamed spec search.template.json to search_template.json
to match the name of the method.
2014-03-21 13:42:52 +01:00
javanna 70c9d6f3bd [TEST] added missing wait for yellow to indices recovery REST test 2014-03-21 10:24:07 +01:00
Honza Král ed842a88da [API] renaming search-template to search_template.
Name of the API should be a valid identifier.
2014-03-20 21:58:08 +01:00
Simon Willnauer bea0dd7c75 [TEST] Fix cat.recovery/10_basic.yaml to allow '-' in the hostname 2014-03-20 20:58:58 +01:00
Simon Willnauer 30532410bf [TEST] disable cat.recovery/10_basic.yaml 2014-03-20 20:43:17 +01:00
Simon Willnauer 6b5001891f [TEST] Added skip clause for regexp feature 2014-03-20 20:41:20 +01:00
Simon Willnauer 9cd3e850af Add simple escape method for special characters to template query
The default mustache engine was using HTML escaping which breaks queries
if used with JSON etc. This commit adds escaping for:

```
\b  Backspace (ascii code 08)
\f  Form feed (ascii code 0C)
\n  New line
\r  Carriage return
\t  Tab
\v  Vertical tab
\"  Double quote
\\  Backslash
```

Closes #5473
2014-03-20 18:48:14 +01:00
Andrew Selden 89e45fde9c Recovery API
Adds a new API endpoint at /_recovery as well as to the Java API. The
recovery API allows one to see the recovery status of all shards in the
cluster. It will report on percent complete, recovery type, and which
files are copied.

Closes #4637
2014-03-20 10:13:30 -07:00
Alexander Reelsen 8f6e1d4720 Query Templates: Adding dedicated /_search/template endpoint
In order to simplify query template execution an own endpoint has been added

Closes #5353
2014-03-20 17:43:40 +01:00
Clinton Gormley 96655d2505 PUT /_aliases should accept a numeric routing value
Also added REST tests for setting index/search/routing
via the PUT /_aliases endpoint

Fixes #5465
2014-03-20 10:38:32 +01:00
Igor Motov a1192044f2 Add ability to get snapshot status for running snapshots
Closes #4946
2014-03-17 20:13:49 -04:00
David Pilato f54e9246c1 Add _cat/plugins endpoint
If we want to have a full picture of versions running in a cluster, we need to add a `_cat/plugins` endpoint.

Response could look like:

```sh
% curl es2:9200/_cat/plugins?v
node component                        version   type url                                   desc
es1  mapper-attachments               1.7.0       j                                        Adds the attachment type allowing to parse difference attachment formats
es1  lang-javascript                  1.4.0       j                                        JavaScript plugin allowing to add javascript scripting support
es1  analysis-smartcn                 1.9.0       j                                        Smart Chinese analysis support
es1  marvel                           1.1.0      j/s http://localhost:9200/_plugins/marvel Elasticsearch Management & Monitoring
es1  kopf                             0.5.3       s  http://localhost:9200/_plugins/kopf   kopf - simple web administration tool for ElasticSearch
es2  mapper-attachments               2.0.0.RC1   j                                        Adds the attachment type allowing to parse difference attachment formats
es2  lang-javascript                  2.0.0.RC1   j                                        JavaScript plugin allowing to add javascript scripting support
es2  analysis-smartcn                 2.0.0.RC1   j                                        Smart Chinese analysis support
```

Closes #4824.
2014-03-16 12:16:09 +01:00
Adrien Grand 65d3b61b97 Add an option to force _optimize operations.
When forced, the index will be merged even if it contains a single segment with
no deletions.

Close #5243
2014-03-14 18:21:56 +01:00
Andrew Selden 673c282abd REST Testing framework enhancement
Adding operators 'lte' and 'gte' to our REST test framework. These
operators test for, respectively, less-than-or-equal and
greater-than-or-equal.
2014-03-10 15:08:43 -07:00
Boaz Leskes b7a95d11a7 Introduced VersionType.FORCE & VersionType.EXTERNAL_GTE
Also added "external_gt" as an alias name for VersionType.EXTERNAL , accessible for the rest layer.

Closes #4213 , Closes #2946
2014-03-10 21:07:17 +01:00
Clinton Gormley f4c0c3dae6 [TESTS] Fixed YAML in put_templates test 2014-03-06 12:16:01 +01:00
James Brook a93d6d55a5 Added support for aliases to index templates
Adapted existing PR (#2739) to updated code (post #4920), added tests and docs (@javanna)

Closes #1825
2014-03-06 11:11:07 +01:00
javanna 7d3cd89207 [TEST] added support for replacing stashed values within objects and lists in our REST tests 2014-03-03 13:34:22 +01:00
Lee Hinman f1f3aa3ac7 [TEST] skip explain reroute REST test until param substitution is implemented 2014-02-28 10:39:07 -07:00
Zachary Tong 0b6c404c51 [TEST] Dynamically extract node ID for test 2014-02-28 08:41:01 -05:00
Zachary Tong 4fe69528d2 [TEST] Add version and master_node parameter to _cluster/state 2014-02-28 08:27:17 -05:00
Lee Hinman e53a43800e Add `explain` flag support to the reroute API
By specifying the `explain` flag, an explanation for the reason a
command can or cannot be executed is returned. No allocation commands
are actually performed.

Returns a response similar to:

{
  "state": {...cluster state...},
  "acknowledged": true,
  "explanations" : [ {
    "command" : "cancel",
      "parameters" : {
        "index" : "decide",
        "shard" : 0,
        "node" : "IvpoKRdtRiGrQ_WKtt4_4w",
        "allow_primary" : false
      },
      "decisions" : [ {
        "decider" : "cancel_allocation_command",
        "decision" : "YES",
        "explanation" : "..."
        } ]
     }, {
      "command" : "move",
      "parameters" : {
        "index" : "decide",
        "shard" : 0,
        "from_node" : "IvpoKRdtRiGrQ_WKtt4_4w",
        "to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
       },
       "decisions" : [ {
         "decider" : "same_shard",
         "decision" : "NO",
         "explanation" : "shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on"
       },
       etc
       ]
  }]
}

also removes AllocationExplanation from cluster state

Closes #2483
Closes #5169
2014-02-27 09:48:51 -07:00
javanna c2bb3981ba [TEST] removed REST test that checks for delete by query shard failures
The clients return an exception in case of failure and not the whole json response containing failures, thus this tests can only work with the Java REST tests runner
2014-02-26 14:11:40 +01:00
javanna e967400522 Expose delete by query shard failures follow-up
- Removed if conditions on index name and shard_id
- Added REST test
2014-02-26 13:14:52 +01:00
Martijn van Groningen 7064056d13 Added TransportGetFieldMappingsIndexAction that uses TransportSingleCustomOperationAction as base class, with the goal to reuse common logic (like: retry on failures, shard picking, connecting to nodes) 2014-02-24 15:54:24 +01:00
Martijn van Groningen a665d514a8 [TEST] Removed whitespace between } and , 2014-02-20 17:45:49 +01:00
Martijn van Groningen 57f40fe691 [TEST] Added whitespaces after brackets, colons and commas in nested percolator test. 2014-02-20 16:00:27 +01:00
Isabel Drost-Fromm 48004ff8a5 Add mustache templating to query execution.
Adds support for storing mustache based query templates that can later be filled
with query parameter values at execution time. Templates may be both quoted,
non-quoted and referencing templates stored in config/scripts/*.mustache by file
name.

See docs/reference/query-dsl/queries/template-query.asciidoc for templating
examples.

Implementation detail: mustache itself is being shaded as it depends directly on
guava - so having it marked optional but included in the final distribution
raises chances of version conflicts downstream.

Fixes #4879
2014-02-20 12:21:59 +01:00
Luca Cavanna 3afdf4a872 Added support for aliases to create index api
It is now possible to specify aliases during index creation:

curl -XPUT 'http://localhost:9200/test' -d '
{
    "aliases" : {
        "alias1" : {},
        "alias2" : {
            "filter" : { "term" : {"field":"value"}}
        }
    }
}'

Closes #4920
2014-02-17 14:54:21 +01:00
Britta Weber db3c6c2a8e Enable percolation for nested documents
closes #5082
2014-02-14 22:42:33 +01:00
Luca Cavanna ac9e2cd46f [TEST] added missing wait for yellow in put warmer REST test 2014-02-14 11:35:40 +01:00
Martijn van Groningen d005a105ac Made highlight query also work in the percolate api. 2014-02-12 10:13:35 -05:00
Luca Cavanna 87a2188ae3 [TEST] disabled replicas in validate query test 2014-02-11 17:16:43 +01:00
Alexander Reelsen b02e6dc996 Migrating NodesInfo API to use plugins instead of singular plugin
In order to be consistent (and because in 1.0 we switched from
parameter driven information to specifzing the metrics as part of the URI)
this patch moves from 'plugin' to 'plugins' in the Nodes Info API.
2014-02-11 10:05:10 +01:00
Luca Cavanna f996b04a9f [TEST] Made cat.count REST test more readable 2014-02-10 16:22:38 +01:00
Luca Cavanna dbc2ba22a8 [TEST] Added REST tests for _cat/thread_pool endpoint 2014-02-10 16:20:03 +01:00
Luca Cavanna 63eccce8f9 [TEST] moved routing REST tests to 0 replicas
REST tests get run against either 1 node or multiple nodes. Wait for yellow with replicas>0 is not enough when running against multiple nodes as replicas shard might get initialized  during testing, which can cause timing issues.
2014-02-10 12:32:34 +01:00
Luca Cavanna a3ad592198 [TEST] Fixed get_source/60_realtime_refresh REST test to work against multiple nodes
Replaced also wait for yellow with wait for green when using no replicas.

get_source/60_realtime_refresh tests per shard refresh using refresh:true and realtime:true in get api. We might run into troubles though if we have a replica that gets initialized after a doc was indexed without a refresh, as that doc will be found when searching against that specific replica shard (as a refresh happens automatically before a replica gets exposed as started).
2014-02-10 12:32:34 +01:00
Simon Willnauer dc308db4fe [TEST] Add basic test for /cat/shards/{index}
Basic REST test for rendering the cat/shards endpoint.
2014-02-10 12:23:01 +01:00
Luca Cavanna e02635e9c2 [TEST] Fixed delete/50_refresh REST test to work against multiple nodes
delete/50_refresh tests per shard refresh using refresh:true in delete api. We might run into troubles though if we have a replica that gets initialized after a doc was indexed and deleted, without a refresh, as that doc won't be found when searching against that specific replica shard (as a refresh happens automatically before a replica gets exposed as started).
2014-02-10 10:58:13 +01:00
Clinton Gormley 8bf302f6e2 [TEST] Removed nested quantifier in cat.allocation regex and made node name regex more liberal 2014-02-09 14:19:59 +01:00
Simon Willnauer f9c19dba56 Wait for green in mlt tests 2014-02-07 19:20:36 +01:00
Clinton Gormley cd679b9e57 [TESTS] Looks like Java character classes require explicit \s 2014-02-07 18:28:00 +01:00
Luca Cavanna 8ef82f9bdf [TEST] fixed cat/allocation REST tests 2014-02-07 18:22:07 +01:00
Clinton Gormley 8d98dbead0 [TESTS] Make cat.allocation tests handle different disk sizes 2014-02-07 17:54:38 +01:00
Clinton Gormley 027a418930 [TESTS] Fixed cat.allocation tests to work with one node cluster 2014-02-07 17:44:41 +01:00
Clinton Gormley 3bd0f6cb63 [TEST] Added cat.allocation and cat.aliases tests 2014-02-07 17:36:01 +01:00
Luca Cavanna c82b231ad8 [TEST] disabled refresh for get_source realtime test 2014-02-07 13:52:49 +01:00
Luca Cavanna 05cc67c872 [TEST] slightly sped up warmer tests by providing warmers during index creation 2014-02-07 13:52:49 +01:00
Luca Cavanna acaa5c7519 [TEST] removed leftover delete alias test in delete mapping 2014-02-07 13:52:49 +01:00
Luca Cavanna 63f9281bbf [TEST] added missing put warmer body 2014-02-03 15:30:58 +01:00
Luca Cavanna c72e1de1c4 [TEST] fixed typo s/mising/missing 2014-02-03 15:30:58 +01:00
Luca Cavanna 88771fdf99 [TEST] Added ability to test apis that don't return json
The last response body gets now always stashed in the REST tests and can be retrieved via `$body`. This implies that not only expected values can be retrieved from the stashed values, but actual values as well.

Added support for regular expressions to `match` assertion, using `Pattern.COMMENTS` flag for better readability through new custom hamcrest matcher (adopted in do section as well). Functionality added through new feature called `regex` that needs to be mentioned in the skip sections whenever needed till all the runners support it.

Added also example tests for cat count api
2014-02-03 11:26:38 +01:00
Luca Cavanna d68d8fbf11 [TEST] clarified skip features usage 2014-01-31 18:03:24 +01:00
Luca Cavanna 33d5a722b3 [TEST] Added ability to skip REST test suite/sections based on their required features
As we have different runners for the REST tests we need a mechanism that allows us to add features to any of them without breaking all others builds.
The idea is to name a feature and temporarily use skip sections that mention the required new features, so that runners that don't support it will skip the test.

Added support for `features` field in skip section.
Added `Features` class that contains a static list of the features supported by the runner. If a feature mentioned in a skip section is not listed here, the test will be skipped.
2014-01-31 16:47:40 +01:00
Clinton Gormley c900ec2152 [TEST] Fixed whitespace in cluster.state/20_filtering.yaml 2014-01-29 13:33:29 +01:00
Alexander Reelsen 5e58f4066e REST API: Consistent get field mapping response
If a get field mapping request is issued, and all but the field can be
found, the response should return an empty JSON object instead of a 404.

Closes #4738
2014-01-28 08:12:53 +01:00
Martijn van Groningen f38296da61 Percolator response now always returns the `matches` key.
Closes #4881
2014-01-27 16:37:09 +01:00
Luca Cavanna da1e3ed8fc [TEST] Added REST tests for create index api 2014-01-27 14:50:04 +01:00
Alexander Reelsen 24abb6cf3f Cluster state toXContent serialization only returns needed data
In order to make sure, that only the requested data is returned to the client,
a couple of fixes have been applied in the ClusterState.toXContent() method.
Also some tests were added to the yaml test suite

Closes #4885
2014-01-27 12:04:57 +01:00
David Pilato 09575eb95f Revert mget yaml test changes
Relative to #4892
2014-01-27 11:28:32 +01:00
David Pilato fdbdb705b9 Revert mget yaml test changes
Relative to #4892
2014-01-27 11:25:39 +01:00
David Pilato 4c50770a89 mget REST API should support source parameter
As stated in documentation, we should support `?source=` parameter in mget REST operations.

This is how to reproduce it:

```sh
curl -XDELETE "http://localhost:9200/test"

curl -XPOST "http://localhost:9200/test/type/1?refresh" -d'{
    "foo": "bar"
}'

curl -XPOST "http://localhost:9200/test/type/_mget" -d'{
    "ids": ["1"]
}'

curl -XGET "http://localhost:9200/test/type/_mget?source=%7B%22ids%22%3A%20%5B%221%22%5D%7D"
```

Closes #4892.
2014-01-27 11:05:50 +01:00
Clinton Gormley d70e6c4730 [SPEC] Updated docs URLs to point to master 2014-01-22 14:40:31 +01:00
Clinton Gormley 882fbb72fd [SPEC] Specify bulk format for body in bulk, msearch, mpercolate 2014-01-21 16:31:51 +01:00
Clinton Gormley 5800209a1e [TEST] Added newline at end of percolate tests 2014-01-21 16:31:50 +01:00
Clinton Gormley acdf2a5825 [SPEC] Renamed termvectors.* to termvector and mtermvectors
and fixed the YAML syntax errors
2014-01-21 16:31:50 +01:00
Britta Weber cae5eb479a [SPEC,TEST,FIX] add spec and tests for termvector api and fix inconsistencies
- index and type were not read from the uri with the _mtermvectors api
- ids were not read from the uri parameters with the _mtermvectors api
2014-01-21 10:03:25 +01:00