From 2e79246c1a9c754a28fb7f96bf572542e7163c3b Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Mon, 13 Jan 2014 21:53:44 +0100 Subject: [PATCH 01/34] [DOCS] Added docs for tribe node Related #4708 --- docs/reference/modules.asciidoc | 2 + docs/reference/modules/tribe.asciidoc | 60 +++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 docs/reference/modules/tribe.asciidoc diff --git a/docs/reference/modules.asciidoc b/docs/reference/modules.asciidoc index a9490ec9101..4ca327d1da0 100644 --- a/docs/reference/modules.asciidoc +++ b/docs/reference/modules.asciidoc @@ -17,6 +17,8 @@ include::modules/network.asciidoc[] include::modules/node.asciidoc[] +include::modules/tribe_node.asciidoc[] + include::modules/plugins.asciidoc[] include::modules/scripting.asciidoc[] diff --git a/docs/reference/modules/tribe.asciidoc b/docs/reference/modules/tribe.asciidoc new file mode 100644 index 00000000000..b7481aec372 --- /dev/null +++ b/docs/reference/modules/tribe.asciidoc @@ -0,0 +1,60 @@ +[[modules-tribe]] +== Tribe node + +The _tribes_ feature allows a _tribe node_ to act as a federated client across +multiple clusters. + +WARNING: This feature is EXPERIMENTAL -- use at your own risk. + +The tribe node works by retrieving the cluster state from all connected +clusters and merging them into a global cluster state. With this information +at hand, it is able to perform read and write operations against the nodes in +all clusters as if they were local. + +The `elasticsearch.yml` config file for a tribe node just needs to list the +clusters that should be joined, for instance: + +[source,yaml] +-------------------------------- +tribe: + t1: <1> + cluster.name: cluster_one + t2: <1> + cluster.name: cluster_two +-------------------------------- +<1> `t1` and `t2` are aribitrary names representing the connection to each + cluster. + +The example above configures connections to two clusters, name `t1` and `t2` +respectively. The tribe node will create a <> to +connect each cluster using <> by default. Any +other settings for the connection can be configured under `tribe.{name}`, just +like the `cluster.name` in the example. + +The merged global cluster state means that almost all operations work in the +same way as a single cluster: distributed search, suggest, percolation, +indexing, etc. + +However, there are a few exceptions: + +* The merged view cannot handle indices with the same name in multiple + clusters. It will pick one of them and discard the other. + +* Master level read operations (eg <>, <>) + need to have the `local` flag set to `true` as the tribe node does not + have a single master node. + +* Master level write operations (eg <>) are not + allowed. These should be performed on a single cluster. + +The tribe node can be configured to block all write operations and all +metadata operations with: + +[source,yaml] +-------------------------------- +tribe: + blocks: + write: true + metadata: true +-------------------------------- + From 0751f0b7c66429382be01bbab9007f043f1adb3c Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Mon, 13 Jan 2014 22:01:12 +0100 Subject: [PATCH 02/34] [DOCS] Fixed link to tribe.asciidoc --- docs/reference/modules.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/modules.asciidoc b/docs/reference/modules.asciidoc index 4ca327d1da0..79150367c1d 100644 --- a/docs/reference/modules.asciidoc +++ b/docs/reference/modules.asciidoc @@ -17,7 +17,7 @@ include::modules/network.asciidoc[] include::modules/node.asciidoc[] -include::modules/tribe_node.asciidoc[] +include::modules/tribe.asciidoc[] include::modules/plugins.asciidoc[] From a1efa1f7aa24631040af8f988135277ff20d2326 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Mon, 13 Jan 2014 21:45:42 +0100 Subject: [PATCH 03/34] Remove `ElasticsearchInterruptedException` and handle interrupt state correctly. InterruptedExceptions should be handled by either rethrowing or restoring the interrupt state (i.e. calling `Thread.currentThread().interrupt()`). This is important since the caller of the is method or subequent method calls might also be interested in this exception. If we ignore the interrupt state the caller might be left unaware of the exception and blocks again on a subsequent method. Closes #4712 --- .../ElasticsearchInterruptedException.java | 36 ------------------- .../elasticsearch/action/ActionFuture.java | 30 ++++++++-------- .../action/support/AdapterActionFuture.java | 8 +++-- .../common/util/concurrent/EsAbortPolicy.java | 4 +-- .../index/service/InternalIndexService.java | 4 +-- .../transport/PlainTransportFuture.java | 8 +++-- .../org/elasticsearch/tribe/TribeService.java | 12 ++++--- 7 files changed, 37 insertions(+), 65 deletions(-) delete mode 100644 src/main/java/org/elasticsearch/ElasticsearchInterruptedException.java diff --git a/src/main/java/org/elasticsearch/ElasticsearchInterruptedException.java b/src/main/java/org/elasticsearch/ElasticsearchInterruptedException.java deleted file mode 100644 index 57a7e20d4e8..00000000000 --- a/src/main/java/org/elasticsearch/ElasticsearchInterruptedException.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to Elasticsearch under one or more contributor - * license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright - * ownership. Elasticsearch licenses this file to you under - * the Apache License, Version 2.0 (the "License"); you may - * not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.elasticsearch; - -/** - * The same as {@link InterruptedException} simply a runtime one. - * - * - */ -public class ElasticsearchInterruptedException extends ElasticsearchException { - - public ElasticsearchInterruptedException(String message) { - super(message); - } - - public ElasticsearchInterruptedException(String message, Throwable cause) { - super(message, cause); - } -} diff --git a/src/main/java/org/elasticsearch/action/ActionFuture.java b/src/main/java/org/elasticsearch/action/ActionFuture.java index 3454aaa8a84..3848536b0d1 100644 --- a/src/main/java/org/elasticsearch/action/ActionFuture.java +++ b/src/main/java/org/elasticsearch/action/ActionFuture.java @@ -34,9 +34,9 @@ import java.util.concurrent.TimeUnit; public interface ActionFuture extends Future { /** - * Similar to {@link #get()}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get()}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is @@ -45,9 +45,9 @@ public interface ActionFuture extends Future { T actionGet() throws ElasticsearchException; /** - * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is @@ -56,9 +56,9 @@ public interface ActionFuture extends Future { T actionGet(String timeout) throws ElasticsearchException; /** - * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is @@ -69,9 +69,9 @@ public interface ActionFuture extends Future { T actionGet(long timeoutMillis) throws ElasticsearchException; /** - * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is @@ -80,9 +80,9 @@ public interface ActionFuture extends Future { T actionGet(long timeout, TimeUnit unit) throws ElasticsearchException; /** - * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just wrapping the {@link InterruptedException} with - * {@link org.elasticsearch.ElasticsearchInterruptedException}, and throwing the actual - * cause of the {@link java.util.concurrent.ExecutionException}. + * Similar to {@link #get(long, java.util.concurrent.TimeUnit)}, just catching the {@link InterruptedException} with + * restoring the interrupted state on the thread and throwing an {@link org.elasticsearch.ElasticsearchIllegalStateException}, + * and throwing the actual cause of the {@link java.util.concurrent.ExecutionException}. *

*

Note, the actual cause is unwrapped to the actual failure (for example, unwrapped * from {@link org.elasticsearch.transport.RemoteTransportException}. The root failure is diff --git a/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java b/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java index 4c8899705aa..f5acc4f0427 100644 --- a/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java +++ b/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java @@ -20,7 +20,7 @@ package org.elasticsearch.action.support; import org.elasticsearch.ElasticsearchException; -import org.elasticsearch.ElasticsearchInterruptedException; +import org.elasticsearch.ElasticsearchIllegalStateException; import org.elasticsearch.ElasticsearchTimeoutException; import org.elasticsearch.action.ActionFuture; import org.elasticsearch.action.ActionListener; @@ -44,7 +44,8 @@ public abstract class AdapterActionFuture extends BaseFuture implements try { return get(); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage()); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Future got interrupted", e); } catch (ExecutionException e) { throw rethrowExecutionException(e); } @@ -72,7 +73,8 @@ public abstract class AdapterActionFuture extends BaseFuture implements } catch (TimeoutException e) { throw new ElasticsearchTimeoutException(e.getMessage()); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage()); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Future got interrupted", e); } catch (ExecutionException e) { throw rethrowExecutionException(e); } diff --git a/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java b/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java index 8c5ee7150fc..cf6239445d8 100644 --- a/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java +++ b/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java @@ -20,7 +20,6 @@ package org.elasticsearch.common.util.concurrent; import org.elasticsearch.ElasticsearchIllegalStateException; -import org.elasticsearch.ElasticsearchInterruptedException; import org.elasticsearch.common.metrics.CounterMetric; import java.util.concurrent.BlockingQueue; @@ -44,7 +43,8 @@ public class EsAbortPolicy implements XRejectedExecutionHandler { try { ((SizeBlockingQueue) queue).forcePut(r); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage(), e); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("forced execution, but got interrupted", e); } return; } diff --git a/src/main/java/org/elasticsearch/index/service/InternalIndexService.java b/src/main/java/org/elasticsearch/index/service/InternalIndexService.java index 88a752fd6e2..130dc912537 100644 --- a/src/main/java/org/elasticsearch/index/service/InternalIndexService.java +++ b/src/main/java/org/elasticsearch/index/service/InternalIndexService.java @@ -24,7 +24,6 @@ import com.google.common.collect.ImmutableSet; import com.google.common.collect.UnmodifiableIterator; import org.elasticsearch.ElasticsearchException; import org.elasticsearch.ElasticsearchIllegalStateException; -import org.elasticsearch.ElasticsearchInterruptedException; import org.elasticsearch.cluster.metadata.IndexMetaData; import org.elasticsearch.common.Nullable; import org.elasticsearch.common.inject.*; @@ -276,7 +275,8 @@ public class InternalIndexService extends AbstractIndexComponent implements Inde try { latch.await(); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException("interrupted closing index [ " + index().name() + "]", e); + logger.debug("Interrupted closing index [{}]", e, index().name()); + Thread.currentThread().interrupt(); } } diff --git a/src/main/java/org/elasticsearch/transport/PlainTransportFuture.java b/src/main/java/org/elasticsearch/transport/PlainTransportFuture.java index 14a1c2c6405..812eec9f64b 100644 --- a/src/main/java/org/elasticsearch/transport/PlainTransportFuture.java +++ b/src/main/java/org/elasticsearch/transport/PlainTransportFuture.java @@ -20,7 +20,7 @@ package org.elasticsearch.transport; import org.elasticsearch.ElasticsearchException; -import org.elasticsearch.ElasticsearchInterruptedException; +import org.elasticsearch.ElasticsearchIllegalStateException; import org.elasticsearch.ElasticsearchTimeoutException; import org.elasticsearch.common.util.concurrent.BaseFuture; @@ -44,7 +44,8 @@ public class PlainTransportFuture extends BaseFutur try { return get(); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage()); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Future got interrupted", e); } catch (ExecutionException e) { if (e.getCause() instanceof ElasticsearchException) { throw (ElasticsearchException) e.getCause(); @@ -61,7 +62,8 @@ public class PlainTransportFuture extends BaseFutur } catch (TimeoutException e) { throw new ElasticsearchTimeoutException(e.getMessage()); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage()); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Future got interrupted", e); } catch (ExecutionException e) { if (e.getCause() instanceof ElasticsearchException) { throw (ElasticsearchException) e.getCause(); diff --git a/src/main/java/org/elasticsearch/tribe/TribeService.java b/src/main/java/org/elasticsearch/tribe/TribeService.java index f28224ef96b..e41700afdd9 100644 --- a/src/main/java/org/elasticsearch/tribe/TribeService.java +++ b/src/main/java/org/elasticsearch/tribe/TribeService.java @@ -22,7 +22,7 @@ package org.elasticsearch.tribe; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Lists; import org.elasticsearch.ElasticsearchException; -import org.elasticsearch.ElasticsearchInterruptedException; +import org.elasticsearch.ElasticsearchIllegalStateException; import org.elasticsearch.cluster.*; import org.elasticsearch.cluster.block.ClusterBlock; import org.elasticsearch.cluster.block.ClusterBlockLevel; @@ -150,8 +150,11 @@ public class TribeService extends AbstractLifecycleComponent { @Override public void onFailure(String source, Throwable t) { - logger.error("{}", t, source); - latch.countDown(); + try { + logger.error("{}", t, source); + } finally { + latch.countDown(); + } } @Override @@ -162,7 +165,8 @@ public class TribeService extends AbstractLifecycleComponent { try { latch.await(); } catch (InterruptedException e) { - throw new ElasticsearchInterruptedException(e.getMessage(), e); + Thread.currentThread().interrupt(); + throw new ElasticsearchIllegalStateException("Interrupted while starting [" + this.getClass().getSimpleName()+ "]", e); } for (InternalNode node : nodes) { try { From b379bf5668c5bf3a6edffbc2cd9ae97f29a8b7d6 Mon Sep 17 00:00:00 2001 From: Lee Hinman Date: Thu, 26 Dec 2013 11:25:25 -0700 Subject: [PATCH 04/34] Default to not accepting type wrapper in indexing requests Currently it is possible to index a document as: ``` POST /myindex/mytype/1 { "foo"...} ``` or as: ``` POST /myindex/mytype/1 { "mytype": { "foo"... } } ``` This makes indexing non-deterministic and fields can be misinterpreted as type names. This changes makes Elasticsearch accept only the first form by default, ie without the type wrapper. This can be changed by setting `index.mapping.allow_type_wrapper` to `true`` when creating the index. Closes #4484 --- docs/reference/docs/index_.asciidoc | 17 +++-- .../index/mapper/DocumentMapper.java | 21 +++--- .../aliases/IndexAliasesTests.java | 2 +- .../mapper/all/SimpleAllMapperTests.java | 4 +- .../elasticsearch/index/mapper/all/test1.json | 34 +++++---- .../mapper/simple/SimpleMapperTests.java | 16 +++++ .../index/mapper/simple/test1-withtype.json | 43 +++++++++++ .../index/mapper/simple/test1.json | 68 +++++++++--------- .../ParseDocumentTypeLevelsTests.java | 54 +++++++------- .../org/elasticsearch/index/query/data.json | 72 +++++++++---------- .../percolator/PercolatorTests.java | 22 ++---- 11 files changed, 200 insertions(+), 153 deletions(-) create mode 100644 src/test/java/org/elasticsearch/index/mapper/simple/test1-withtype.json diff --git a/docs/reference/docs/index_.asciidoc b/docs/reference/docs/index_.asciidoc index c193cbdeaaf..de062c80ff3 100644 --- a/docs/reference/docs/index_.asciidoc +++ b/docs/reference/docs/index_.asciidoc @@ -33,7 +33,7 @@ The result of the above index operation is: The index operation automatically creates an index if it has not been created before (check out the -<> for manually +<> for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if one has not yet been created (check out the <> @@ -44,12 +44,21 @@ objects will automatically be added to the mapping definition of the type specified. Check out the <> section for more information on mapping definitions. -Though explained on the <> section, -it's important to note that the format of the JSON document can also -include the type (very handy when using JSON mappers), for example: +Note that the format of the JSON document can also include the type (very handy +when using JSON mappers) if the `index.mapping.allow_type_wrapper` setting is +set to true, for example: [source,js] -------------------------------------------------- +$ curl -XPOST 'http://localhost:9200/twitter' -d '{ + "settings": { + "index": { + "mapping.allow_type_wrapper": true + } + } +}' +{"acknowledged":true} + $ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{ "tweet" : { "user" : "kimchy", diff --git a/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java b/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java index d2745d51363..98157f28870 100644 --- a/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java +++ b/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java @@ -239,6 +239,8 @@ public class DocumentMapper implements ToXContent { } }; + public static final String ALLOW_TYPE_WRAPPER = "index.mapping.allow_type_wrapper"; + private final String index; private final Settings indexSettings; @@ -494,18 +496,15 @@ public class DocumentMapper implements ToXContent { } else if (token != XContentParser.Token.FIELD_NAME) { throw new MapperParsingException("Malformed content, after first object, either the type field or the actual properties should exist"); } - if (type.equals(parser.currentName())) { - // first field is the same as the type, this might be because the type is provided, and the object exists within it - // or because there is a valid field that by chance is named as the type - - // Note, in this case, we only handle plain value types, an object type will be analyzed as if it was the type itself - // and other same level fields will be ignored - token = parser.nextToken(); + // first field is the same as the type, this might be because the + // type is provided, and the object exists within it or because + // there is a valid field that by chance is named as the type. + // Because of this, by default wrapping a document in a type is + // disabled, but can be enabled by setting + // index.mapping.allow_type_wrapper to true + if (type.equals(parser.currentName()) && indexSettings.getAsBoolean(ALLOW_TYPE_WRAPPER, false)) { + parser.nextToken(); countDownTokens++; - // commented out, allow for same type with START_OBJECT, we do our best to handle it except for the above corner case -// if (token != XContentParser.Token.START_OBJECT) { -// throw new MapperException("Malformed content, a field with the same name as the type must be an object with the properties/fields within it"); -// } } for (RootMapper rootMapper : rootMappersOrdered) { diff --git a/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java b/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java index 34653c9c35b..5599857fd18 100644 --- a/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java +++ b/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java @@ -853,6 +853,6 @@ public class IndexAliasesTests extends ElasticsearchIntegrationTest { } private String source(String id, String nameValue) { - return "{ type1 : { \"id\" : \"" + id + "\", \"name\" : \"" + nameValue + "\" } }"; + return "{ \"id\" : \"" + id + "\", \"name\" : \"" + nameValue + "\" }"; } } diff --git a/src/test/java/org/elasticsearch/index/mapper/all/SimpleAllMapperTests.java b/src/test/java/org/elasticsearch/index/mapper/all/SimpleAllMapperTests.java index 0d8d8bebd76..1f15caf8618 100644 --- a/src/test/java/org/elasticsearch/index/mapper/all/SimpleAllMapperTests.java +++ b/src/test/java/org/elasticsearch/index/mapper/all/SimpleAllMapperTests.java @@ -231,11 +231,11 @@ public class SimpleAllMapperTests extends ElasticsearchTestCase { // reparse it DocumentMapper builtDocMapper = MapperTestUtils.newParser().parse(builtMapping); - byte[] json = jsonBuilder().startObject().startObject("test") + byte[] json = jsonBuilder().startObject() .field("foo", "bar") .field("_id", 1) .field("foobar", "foobar") - .endObject().endObject().bytes().array(); + .endObject().bytes().array(); Document doc = builtDocMapper.parse(new BytesArray(json)).rootDoc(); AllField field = (AllField) doc.getField("_all"); if (enabled) { diff --git a/src/test/java/org/elasticsearch/index/mapper/all/test1.json b/src/test/java/org/elasticsearch/index/mapper/all/test1.json index c3e72dfa9a8..834400a48cb 100644 --- a/src/test/java/org/elasticsearch/index/mapper/all/test1.json +++ b/src/test/java/org/elasticsearch/index/mapper/all/test1.json @@ -1,20 +1,18 @@ { - "person":{ - "_boost":3.7, - "_id":"1", - "name":{ - "first":"shay", - "last":"banon" + "_boost":3.7, + "_id":"1", + "name":{ + "first":"shay", + "last":"banon" + }, + "address":{ + "first":{ + "location":"first location" }, - "address":{ - "first":{ - "location":"first location" - }, - "last":{ - "location":"last location" - } - }, - "simple1":1, - "simple2":2 - } -} \ No newline at end of file + "last":{ + "location":"last location" + } + }, + "simple1":1, + "simple2":2 +} diff --git a/src/test/java/org/elasticsearch/index/mapper/simple/SimpleMapperTests.java b/src/test/java/org/elasticsearch/index/mapper/simple/SimpleMapperTests.java index dd7291b22bb..79951e46a96 100644 --- a/src/test/java/org/elasticsearch/index/mapper/simple/SimpleMapperTests.java +++ b/src/test/java/org/elasticsearch/index/mapper/simple/SimpleMapperTests.java @@ -22,6 +22,8 @@ package org.elasticsearch.index.mapper.simple; import com.google.common.base.Charsets; import org.elasticsearch.common.bytes.BytesArray; import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; import org.elasticsearch.index.mapper.*; import org.elasticsearch.index.mapper.ParseContext.Document; import org.elasticsearch.test.ElasticsearchTestCase; @@ -128,4 +130,18 @@ public class SimpleMapperTests extends ElasticsearchTestCase { assertThat(e.getMessage(), equalTo("failed to parse, document is empty")); } } + + @Test + public void testTypeWrapperWithSetting() throws Exception { + String mapping = copyToStringFromClasspath("/org/elasticsearch/index/mapper/simple/test-mapping.json"); + Settings settings = ImmutableSettings.settingsBuilder().put("index.mapping.allow_type_wrapper", true).build(); + DocumentMapper docMapper = MapperTestUtils.newParser(settings).parse(mapping); + + assertThat((String) docMapper.meta().get("param1"), equalTo("value1")); + + BytesReference json = new BytesArray(copyToBytesFromClasspath("/org/elasticsearch/index/mapper/simple/test1-withtype.json")); + Document doc = docMapper.parse(json).rootDoc(); + assertThat(doc.get(docMapper.uidMapper().names().indexName()), equalTo(Uid.createUid("person", "1"))); + assertThat(doc.get(docMapper.mappers().name("first").mapper().names().indexName()), equalTo("shay")); + } } diff --git a/src/test/java/org/elasticsearch/index/mapper/simple/test1-withtype.json b/src/test/java/org/elasticsearch/index/mapper/simple/test1-withtype.json new file mode 100644 index 00000000000..096554abb20 --- /dev/null +++ b/src/test/java/org/elasticsearch/index/mapper/simple/test1-withtype.json @@ -0,0 +1,43 @@ +{ + person:{ + _boost:3.7, + _id:"1", + name:{ + first:"shay", + last:"banon" + }, + address:{ + first:{ + location:"first location" + }, + last:{ + location:"last location" + } + }, + age:32, + birthDate:"1977-11-15", + nerd:true, + dogs:["buck", "mia"], + complex:[ + { + value1:"value1" + }, + { + value2:"value2" + } + ], + complex2:[ + [ + { + value1:"value1" + } + ], + [ + { + value2:"value2" + } + ] + ], + nullValue:null + } +} diff --git a/src/test/java/org/elasticsearch/index/mapper/simple/test1.json b/src/test/java/org/elasticsearch/index/mapper/simple/test1.json index 0d678692999..93507daffef 100644 --- a/src/test/java/org/elasticsearch/index/mapper/simple/test1.json +++ b/src/test/java/org/elasticsearch/index/mapper/simple/test1.json @@ -1,43 +1,41 @@ { - person:{ - _boost:3.7, - _id:"1", - name:{ - first:"shay", - last:"banon" + _boost:3.7, + _id:"1", + name:{ + first:"shay", + last:"banon" + }, + address:{ + first:{ + location:"first location" }, - address:{ - first:{ - location:"first location" - }, - last:{ - location:"last location" - } + last:{ + location:"last location" + } + }, + age:32, + birthDate:"1977-11-15", + nerd:true, + dogs:["buck", "mia"], + complex:[ + { + value1:"value1" }, - age:32, - birthDate:"1977-11-15", - nerd:true, - dogs:["buck", "mia"], - complex:[ + { + value2:"value2" + } + ], + complex2:[ + [ { value1:"value1" - }, + } + ], + [ { value2:"value2" } - ], - complex2:[ - [ - { - value1:"value1" - } - ], - [ - { - value2:"value2" - } - ] - ], - nullValue:null - } -} \ No newline at end of file + ] + ], + nullValue:null +} diff --git a/src/test/java/org/elasticsearch/index/mapper/typelevels/ParseDocumentTypeLevelsTests.java b/src/test/java/org/elasticsearch/index/mapper/typelevels/ParseDocumentTypeLevelsTests.java index 2bde20e14f2..56d1a961c8d 100644 --- a/src/test/java/org/elasticsearch/index/mapper/typelevels/ParseDocumentTypeLevelsTests.java +++ b/src/test/java/org/elasticsearch/index/mapper/typelevels/ParseDocumentTypeLevelsTests.java @@ -26,9 +26,7 @@ import org.elasticsearch.index.mapper.ParsedDocument; import org.elasticsearch.test.ElasticsearchTestCase; import org.junit.Test; -import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.equalTo; -import static org.hamcrest.Matchers.nullValue; /** * @@ -68,9 +66,9 @@ public class ParseDocumentTypeLevelsTests extends ElasticsearchTestCase { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -109,10 +107,10 @@ public class ParseDocumentTypeLevelsTests extends ElasticsearchTestCase { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type"), equalTo("value_type")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type"), equalTo("value_type")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -131,9 +129,9 @@ public class ParseDocumentTypeLevelsTests extends ElasticsearchTestCase { .bytes()); // in this case, we analyze the type object as the actual document, and ignore the other same level fields - assertThat(doc.rootDoc().get("type_field"), equalTo("type_value")); - assertThat(doc.rootDoc().get("test1"), nullValue()); - assertThat(doc.rootDoc().get("test2"), nullValue()); + assertThat(doc.rootDoc().get("type.type_field"), equalTo("type_value")); + assertThat(doc.rootDoc().get("test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("test2"), equalTo("value2")); } @Test @@ -151,10 +149,10 @@ public class ParseDocumentTypeLevelsTests extends ElasticsearchTestCase { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type.type_field"), equalTo("type_value")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type.type_field"), equalTo("type_value")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -172,10 +170,10 @@ public class ParseDocumentTypeLevelsTests extends ElasticsearchTestCase { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type"), equalTo("value_type")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type"), equalTo("value_type")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -193,10 +191,10 @@ public class ParseDocumentTypeLevelsTests extends ElasticsearchTestCase { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type"), equalTo("value_type")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type"), equalTo("value_type")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } @Test @@ -236,9 +234,9 @@ public class ParseDocumentTypeLevelsTests extends ElasticsearchTestCase { .endObject().endObject() .bytes()); - assertThat(doc.rootDoc().get("type.type_field"), equalTo("type_value")); - assertThat(doc.rootDoc().get("test1"), equalTo("value1")); - assertThat(doc.rootDoc().get("test2"), equalTo("value2")); - assertThat(doc.rootDoc().get("inner.inner_field"), equalTo("inner_value")); + assertThat(doc.rootDoc().get("type.type.type_field"), equalTo("type_value")); + assertThat(doc.rootDoc().get("type.test1"), equalTo("value1")); + assertThat(doc.rootDoc().get("type.test2"), equalTo("value2")); + assertThat(doc.rootDoc().get("type.inner.inner_field"), equalTo("inner_value")); } } diff --git a/src/test/java/org/elasticsearch/index/query/data.json b/src/test/java/org/elasticsearch/index/query/data.json index ef942d9c55d..b3c6db83a07 100644 --- a/src/test/java/org/elasticsearch/index/query/data.json +++ b/src/test/java/org/elasticsearch/index/query/data.json @@ -1,47 +1,45 @@ { - person:{ - _boost:3.7, - _id:"1", - name:{ - first:"shay", - last:"banon" + _boost:3.7, + _id:"1", + name:{ + first:"shay", + last:"banon" + }, + address:{ + first:{ + location:"first location" }, - address:{ - first:{ - location:"first location" - }, - last:{ - location:"last location" - } + last:{ + location:"last location" + } + }, + age:32, + birthDate:"1977-11-15", + nerd:true, + dogs:["buck", "mia"], + complex:[ + { + value1:"value1" }, - age:32, - birthDate:"1977-11-15", - nerd:true, - dogs:["buck", "mia"], - complex:[ + { + value2:"value2" + } + ], + complex2:[ + [ { value1:"value1" - }, + } + ], + [ { value2:"value2" } - ], - complex2:[ - [ - { - value1:"value1" - } - ], - [ - { - value2:"value2" - } - ] - ], - nullValue:null, - "location":{ - "lat":1.1, - "lon":1.2 - } + ] + ], + nullValue:null, + "location":{ + "lat":1.1, + "lon":1.2 } } \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/percolator/PercolatorTests.java b/src/test/java/org/elasticsearch/percolator/PercolatorTests.java index 6c20a50cfe2..42859d75913 100644 --- a/src/test/java/org/elasticsearch/percolator/PercolatorTests.java +++ b/src/test/java/org/elasticsearch/percolator/PercolatorTests.java @@ -164,11 +164,6 @@ public class PercolatorTests extends ElasticsearchIntegrationTest { .field("field2", "value") .endObject().endObject(); - XContentBuilder docWithType = XContentFactory.jsonBuilder().startObject().startObject("doc").startObject("type1") - .field("field1", 1) - .field("field2", "value") - .endObject().endObject().endObject(); - PercolateResponse response = client().preparePercolate().setSource(doc) .setIndices("test").setDocumentType("type1") .execute().actionGet(); @@ -187,13 +182,6 @@ public class PercolatorTests extends ElasticsearchIntegrationTest { assertThat(response.getMatches(), arrayWithSize(1)); assertThat(convertFromTextArray(response.getMatches(), "test"), arrayContaining("test1")); - response = client().preparePercolate() - .setIndices("test").setDocumentType("type1") - .setSource(docWithType).execute().actionGet(); - assertMatchCount(response, 1l); - assertThat(response.getMatches(), arrayWithSize(1)); - assertThat(convertFromTextArray(response.getMatches(), "test"), arrayContaining("test1")); - // add second query... client().prepareIndex("test", PercolatorService.TYPE_NAME, "test2") .setSource(XContentFactory.jsonBuilder().startObject().field("query", termQuery("field1", 1)).endObject()) @@ -430,7 +418,7 @@ public class PercolatorTests extends ElasticsearchIntegrationTest { percolate = client().preparePercolate() .setIndices("test").setDocumentType("type1") - .setSource(jsonBuilder().startObject().startObject("doc").startObject("type1").field("field1", "value2").endObject().endObject().endObject()) + .setSource(jsonBuilder().startObject().startObject("doc").field("field1", "value2").endObject().endObject()) .execute().actionGet(); assertMatchCount(percolate, 1l); assertThat(percolate.getMatches(), arrayWithSize(1)); @@ -471,7 +459,7 @@ public class PercolatorTests extends ElasticsearchIntegrationTest { percolate = client().preparePercolate() .setIndices("test").setDocumentType("type1") - .setSource(jsonBuilder().startObject().startObject("doc").startObject("type1").field("field1", "value2").endObject().endObject().endObject()) + .setSource(jsonBuilder().startObject().startObject("doc").field("field1", "value2").endObject().endObject()) .execute().actionGet(); assertMatchCount(percolate, 1l); assertThat(percolate.getMatches(), arrayWithSize(1)); @@ -487,7 +475,7 @@ public class PercolatorTests extends ElasticsearchIntegrationTest { .execute().actionGet(); PercolateSourceBuilder sourceBuilder = new PercolateSourceBuilder() - .setDoc(docBuilder().setDoc(jsonBuilder().startObject().startObject("type1").field("field1", "value2").endObject().endObject())) + .setDoc(docBuilder().setDoc(jsonBuilder().startObject().field("field1", "value2").endObject())) .setQueryBuilder(termQuery("color", "red")); percolate = client().preparePercolate() .setIndices("test").setDocumentType("type1") @@ -533,9 +521,9 @@ public class PercolatorTests extends ElasticsearchIntegrationTest { logger.info("--> percolate a document"); PercolateResponse percolate = client().preparePercolate().setIndices("test").setDocumentType("type1") .setSource(jsonBuilder().startObject() - .startObject("doc").startObject("type1") + .startObject("doc") .field("field1", "value1") - .endObject().endObject() + .endObject() .endObject()) .execute().actionGet(); assertMatchCount(percolate, 1l); From b987615f5e96cb6f568e025a449d1f4d7e5402df Mon Sep 17 00:00:00 2001 From: Igor Motov Date: Mon, 13 Jan 2014 11:22:13 -0500 Subject: [PATCH 05/34] Improve support for partial snapshots Fixes #4701. Changes behavior of the snapshot operation. The operation now fails if not all primary shards are available at the beginning of the snapshot operation. The restore operation no longer tries to restore indices with shards that failed or were missing during snapshot operation. --- docs/reference/modules/snapshots.asciidoc | 3 +- .../create/CreateSnapshotRequest.java | 43 +++++++-- .../create/CreateSnapshotRequestBuilder.java | 11 +++ .../create/TransportCreateSnapshotAction.java | 1 + .../restore/RestoreSnapshotRequest.java | 13 ++- .../cluster/metadata/SnapshotMetaData.java | 43 ++++++++- .../snapshots/RestoreService.java | 29 +++++- .../elasticsearch/snapshots/SnapshotInfo.java | 24 ++++- .../snapshots/SnapshotsService.java | 88 ++++++++++++++++--- .../snapshots/AbstractSnapshotTests.java | 30 ------- .../DedicatedClusterSnapshotRestoreTests.java | 76 +++++++++++++++- .../snapshots/RepositoriesTests.java | 2 + .../SharedClusterSnapshotRestoreTests.java | 9 +- .../test/ElasticsearchIntegrationTest.java | 22 +++++ 14 files changed, 320 insertions(+), 74 deletions(-) diff --git a/docs/reference/modules/snapshots.asciidoc b/docs/reference/modules/snapshots.asciidoc index 08aab370357..94bf40c4ab5 100644 --- a/docs/reference/modules/snapshots.asciidoc +++ b/docs/reference/modules/snapshots.asciidoc @@ -101,7 +101,8 @@ supports <>. The snapshot request al `ignore_unavailable` option. Setting it to `true` will cause indices that do not exists to be ignored during snapshot creation. By default, when `ignore_unavailable` option is not set and an index is missing the snapshot request will fail. By setting `include_global_state` to false it's possible to prevent the cluster global state to be stored as part of -the snapshot. +the snapshot. By default, entire snapshot will fail if one or more indices participating in the snapshot don't have +all primary shards available. This behaviour can be changed by setting `partial` to `true`. The index snapshot process is incremental. In the process of making the index snapshot Elasticsearch analyses the list of the index files that are already stored in the repository and copies only files that were created or diff --git a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequest.java b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequest.java index 8100b6f2723..2030c3e2383 100644 --- a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/CreateSnapshotRequest.java @@ -45,6 +45,7 @@ import static org.elasticsearch.common.Strings.hasLength; import static org.elasticsearch.common.settings.ImmutableSettings.Builder.EMPTY_SETTINGS; import static org.elasticsearch.common.settings.ImmutableSettings.readSettingsFromStream; import static org.elasticsearch.common.settings.ImmutableSettings.writeSettingsToStream; +import static org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue; /** * Create snapshot request @@ -70,6 +71,8 @@ public class CreateSnapshotRequest extends MasterNodeOperationRequest diff --git a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java index 61f0db8aef0..5b54e414031 100644 --- a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java +++ b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/create/TransportCreateSnapshotAction.java @@ -78,6 +78,7 @@ public class TransportCreateSnapshotAction extends TransportMasterNodeOperationA new SnapshotsService.SnapshotRequest("create_snapshot[" + request.snapshot() + "]", request.snapshot(), request.repository()) .indices(request.indices()) .indicesOptions(request.indicesOptions()) + .partial(request.partial()) .settings(request.settings()) .includeGlobalState(request.includeGlobalState()) .masterNodeTimeout(request.masterNodeTimeout()); diff --git a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java index f5ba01cf650..b009e76a91b 100644 --- a/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java @@ -44,6 +44,7 @@ import static org.elasticsearch.common.Strings.hasLength; import static org.elasticsearch.common.settings.ImmutableSettings.Builder.EMPTY_SETTINGS; import static org.elasticsearch.common.settings.ImmutableSettings.readSettingsFromStream; import static org.elasticsearch.common.settings.ImmutableSettings.writeSettingsToStream; +import static org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue; /** * Restore snapshot request @@ -397,17 +398,13 @@ public class RestoreSnapshotRequest extends MasterNodeOperationRequest shards = ImmutableMap.builder(); for (Map.Entry indexEntry : renamedIndices.entrySet()) { String index = indexEntry.getValue(); + // Make sure that index was fully snapshotted - don't restore + if (failed(snapshot, index)) { + throw new SnapshotRestoreException(snapshotId, "index [" + index + "] wasn't fully snapshotted - cannot restore"); + } RestoreSource restoreSource = new RestoreSource(snapshotId, index); String renamedIndex = indexEntry.getKey(); IndexMetaData snapshotIndexMetaData = metaData.index(index); @@ -391,6 +395,24 @@ public class RestoreService extends AbstractComponent implements ClusterStateLis } } + private boolean failed(Snapshot snapshot, String index) { + for (SnapshotShardFailure failure : snapshot.shardFailures()) { + if (index.equals(failure.index())) { + return true; + } + } + return false; + } + + private boolean failed(Snapshot snapshot, String index, int shard) { + for (SnapshotShardFailure failure : snapshot.shardFailures()) { + if (index.equals(failure.index()) && shard == failure.shardId()) { + return true; + } + } + return false; + } + /** * Adds restore completion listener *

@@ -427,16 +449,17 @@ public class RestoreService extends AbstractComponent implements ClusterStateLis /** * Checks if a repository is currently in use by one of the snapshots + * * @param clusterState cluster state - * @param repository repository id + * @param repository repository id * @return true if repository is currently in use by one of the running snapshots */ public static boolean isRepositoryInUse(ClusterState clusterState, String repository) { MetaData metaData = clusterState.metaData(); RestoreMetaData snapshots = metaData.custom(RestoreMetaData.TYPE); if (snapshots != null) { - for(RestoreMetaData.Entry snapshot : snapshots.entries()) { - if(repository.equals(snapshot.snapshotId().getRepository())) { + for (RestoreMetaData.Entry snapshot : snapshots.entries()) { + if (repository.equals(snapshot.snapshotId().getRepository())) { return true; } } diff --git a/src/main/java/org/elasticsearch/snapshots/SnapshotInfo.java b/src/main/java/org/elasticsearch/snapshots/SnapshotInfo.java index ebf05ab6d87..768a0157945 100644 --- a/src/main/java/org/elasticsearch/snapshots/SnapshotInfo.java +++ b/src/main/java/org/elasticsearch/snapshots/SnapshotInfo.java @@ -24,7 +24,6 @@ import org.elasticsearch.common.io.stream.StreamOutput; import org.elasticsearch.common.io.stream.Streamable; import org.elasticsearch.common.joda.FormatDateTimeFormatter; import org.elasticsearch.common.joda.Joda; -import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.common.xcontent.ToXContent; import org.elasticsearch.common.xcontent.XContentBuilder; import org.elasticsearch.common.xcontent.XContentBuilderString; @@ -43,6 +42,8 @@ public class SnapshotInfo implements ToXContent, Streamable { private SnapshotState state; + private String reason; + private ImmutableList indices; private long startTime; @@ -67,6 +68,7 @@ public class SnapshotInfo implements ToXContent, Streamable { public SnapshotInfo(Snapshot snapshot) { name = snapshot.name(); state = snapshot.state(); + reason = snapshot.reason(); indices = snapshot.indices(); startTime = snapshot.startTime(); endTime = snapshot.endTime(); @@ -93,6 +95,15 @@ public class SnapshotInfo implements ToXContent, Streamable { return state; } + /** + * Returns snapshot failure reason + * + * @return snapshot failure reason + */ + public String reason() { + return reason; + } + /** * Returns indices that were included into this snapshot * @@ -137,7 +148,7 @@ public class SnapshotInfo implements ToXContent, Streamable { * @return number of failed shards */ public int failedShards() { - return totalShards - successfulShards; + return totalShards - successfulShards; } /** @@ -162,6 +173,9 @@ public class SnapshotInfo implements ToXContent, Streamable { * Returns snapshot REST status */ public RestStatus status() { + if (state == SnapshotState.FAILED) { + return RestStatus.INTERNAL_SERVER_ERROR; + } if (shardFailures.size() == 0) { return RestStatus.OK; } @@ -179,6 +193,7 @@ public class SnapshotInfo implements ToXContent, Streamable { static final class Fields { static final XContentBuilderString INDICES = new XContentBuilderString("indices"); static final XContentBuilderString STATE = new XContentBuilderString("state"); + static final XContentBuilderString REASON = new XContentBuilderString("reason"); static final XContentBuilderString START_TIME = new XContentBuilderString("start_time"); static final XContentBuilderString START_TIME_IN_MILLIS = new XContentBuilderString("start_time_in_millis"); static final XContentBuilderString END_TIME = new XContentBuilderString("end_time"); @@ -202,6 +217,9 @@ public class SnapshotInfo implements ToXContent, Streamable { } builder.endArray(); builder.field(Fields.STATE, state); + if (reason != null) { + builder.field(Fields.REASON, reason); + } if (startTime != 0) { builder.field(Fields.START_TIME, DATE_TIME_FORMATTER.printer().print(startTime)); builder.field(Fields.START_TIME_IN_MILLIS, startTime); @@ -235,6 +253,7 @@ public class SnapshotInfo implements ToXContent, Streamable { } indices = indicesListBuilder.build(); state = SnapshotState.fromValue(in.readByte()); + reason = in.readOptionalString(); startTime = in.readVLong(); endTime = in.readVLong(); totalShards = in.readVInt(); @@ -259,6 +278,7 @@ public class SnapshotInfo implements ToXContent, Streamable { out.writeString(index); } out.writeByte(state.value()); + out.writeOptionalString(reason); out.writeVLong(startTime); out.writeVLong(endTime); out.writeVInt(totalShards); diff --git a/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java b/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java index b5f398173c7..641d5c2a255 100644 --- a/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java +++ b/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java @@ -59,6 +59,7 @@ import java.util.concurrent.CopyOnWriteArrayList; import static com.google.common.collect.Lists.newArrayList; import static com.google.common.collect.Maps.newHashMap; import static com.google.common.collect.Maps.newHashMapWithExpectedSize; +import static com.google.common.collect.Sets.newHashSet; /** * Service responsible for creating snapshots @@ -67,7 +68,7 @@ import static com.google.common.collect.Maps.newHashMapWithExpectedSize; *

    *
  • On the master node the {@link #createSnapshot(SnapshotRequest, CreateSnapshotListener)} is called and makes sure that no snapshots is currently running * and registers the new snapshot in cluster state
    • - *
  • When cluster state is updated the {@link #beginSnapshot(ClusterState, SnapshotMetaData.Entry, CreateSnapshotListener)} method + *
  • When cluster state is updated the {@link #beginSnapshot(ClusterState, SnapshotMetaData.Entry, boolean, CreateSnapshotListener)} method * kicks in and initializes the snapshot in the repository and then populates list of shards that needs to be snapshotted in cluster state
    • *
  • Each data node is watching for these shards and when new shards scheduled for snapshotting appear in the cluster state, data nodes * start processing them through {@link #processIndexShardSnapshots(SnapshotMetaData)} method
    • @@ -187,7 +188,7 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL threadPool.executor(ThreadPool.Names.SNAPSHOT).execute(new Runnable() { @Override public void run() { - beginSnapshot(newState, newSnapshot, listener); + beginSnapshot(newState, newSnapshot, request.partial, listener); } }); } @@ -243,9 +244,10 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL * * @param clusterState cluster state * @param snapshot snapshot meta data + * @param partial allow partial snapshots * @param userCreateSnapshotListener listener */ - private void beginSnapshot(ClusterState clusterState, final SnapshotMetaData.Entry snapshot, final CreateSnapshotListener userCreateSnapshotListener) { + private void beginSnapshot(ClusterState clusterState, final SnapshotMetaData.Entry snapshot, final boolean partial, final CreateSnapshotListener userCreateSnapshotListener) { boolean snapshotCreated = false; try { Repository repository = repositoriesService.repository(snapshot.snapshotId().getRepository()); @@ -271,6 +273,7 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL clusterService.submitStateUpdateTask("update_snapshot [" + snapshot + "]", new ProcessedClusterStateUpdateTask() { boolean accepted = false; SnapshotMetaData.Entry updatedSnapshot; + String failure = null; @Override public ClusterState execute(ClusterState currentState) { @@ -282,6 +285,15 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL if (entry.snapshotId().equals(snapshot.snapshotId())) { // Replace the snapshot that was just created ImmutableMap shards = shards(snapshot.snapshotId(), currentState, snapshot.indices()); + if (!partial) { + Set indicesWithMissingShards = indicesWithMissingShards(shards); + if (indicesWithMissingShards != null) { + updatedSnapshot = new SnapshotMetaData.Entry(snapshot.snapshotId(), snapshot.includeGlobalState(), State.FAILED, snapshot.indices(), shards); + entries.add(updatedSnapshot); + failure = "Indices don't have primary shards +[" + indicesWithMissingShards + "]"; + continue; + } + } updatedSnapshot = new SnapshotMetaData.Entry(snapshot.snapshotId(), snapshot.includeGlobalState(), State.STARTED, snapshot.indices(), shards); entries.add(updatedSnapshot); if (!completed(shards.values())) { @@ -310,8 +322,11 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL userCreateSnapshotListener.onResponse(); // Now that snapshot completion listener is registered we can end the snapshot if needed + // We should end snapshot only if 1) we didn't accept it for processing (which happens when there + // is nothing to do) and 2) there was a snapshot in metadata that we should end. Otherwise we should + // go ahead and continue working on this snapshot rather then end here. if (!accepted && updatedSnapshot != null) { - endSnapshot(updatedSnapshot); + endSnapshot(updatedSnapshot, failure); } } }); @@ -602,6 +617,25 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL return true; } + /** + * Returns list of indices with missing shards + * + * @param shards list of shard statuses + * @return list of failed indices + */ + private Set indicesWithMissingShards(ImmutableMap shards) { + Set indices = null; + for (ImmutableMap.Entry entry : shards.entrySet()) { + if (entry.getValue().state() == State.MISSING) { + if (indices == null) { + indices = newHashSet(); + } + indices.add(entry.getKey().getIndex()); + } + } + return indices; + } + /** * Updates the shard status on master node * @@ -661,25 +695,38 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL * * @param entry snapshot */ - private void endSnapshot(final SnapshotMetaData.Entry entry) { + private void endSnapshot(SnapshotMetaData.Entry entry) { + endSnapshot(entry, null); + } + + + /** + * Finalizes the shard in repository and then removes it from cluster state + *

    + * This is non-blocking method that runs on a thread from SNAPSHOT thread pool + * + * @param entry snapshot + * @param failure failure reason or null if snapshot was successful + */ + private void endSnapshot(final SnapshotMetaData.Entry entry, final String failure) { threadPool.executor(ThreadPool.Names.SNAPSHOT).execute(new Runnable() { @Override public void run() { SnapshotId snapshotId = entry.snapshotId(); try { final Repository repository = repositoriesService.repository(snapshotId.getRepository()); - logger.trace("[{}] finalizing snapshot in repository", snapshotId); + logger.trace("[{}] finalizing snapshot in repository, state: [{}], failure[{}]", snapshotId, entry.state(), failure); ArrayList failures = newArrayList(); ArrayList shardFailures = newArrayList(); for (Map.Entry shardStatus : entry.shards().entrySet()) { ShardId shardId = shardStatus.getKey(); ShardSnapshotStatus status = shardStatus.getValue(); - if (status.state() == State.FAILED) { + if (status.state().failed()) { failures.add(new ShardSearchFailure(status.reason(), new SearchShardTarget(status.nodeId(), shardId.getIndex(), shardId.id()))); shardFailures.add(new SnapshotShardFailure(status.nodeId(), shardId.getIndex(), shardId.id(), status.reason())); } } - Snapshot snapshot = repository.finalizeSnapshot(snapshotId, null, entry.shards().size(), ImmutableList.copyOf(shardFailures)); + Snapshot snapshot = repository.finalizeSnapshot(snapshotId, failure, entry.shards().size(), ImmutableList.copyOf(shardFailures)); removeSnapshotFromClusterState(snapshotId, new SnapshotInfo(snapshot), null); } catch (Throwable t) { logger.warn("[{}] failed to finalize snapshot", t, snapshotId); @@ -841,16 +888,17 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL /** * Checks if a repository is currently in use by one of the snapshots + * * @param clusterState cluster state - * @param repository repository id + * @param repository repository id * @return true if repository is currently in use by one of the running snapshots */ public static boolean isRepositoryInUse(ClusterState clusterState, String repository) { MetaData metaData = clusterState.metaData(); SnapshotMetaData snapshots = metaData.custom(SnapshotMetaData.TYPE); if (snapshots != null) { - for(SnapshotMetaData.Entry snapshot : snapshots.entries()) { - if(repository.equals(snapshot.snapshotId().getRepository())) { + for (SnapshotMetaData.Entry snapshot : snapshots.entries()) { + if (repository.equals(snapshot.snapshotId().getRepository())) { return true; } } @@ -900,10 +948,9 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL ShardId shardId = new ShardId(index, i); ShardRouting primary = indexRoutingTable.shard(i).primaryShard(); if (primary == null || !primary.assignedToNode()) { - //TODO: Should we bailout completely or just mark this shard as failed? - builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(null, State.FAILED, "primary shard is not allocated")); + builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(null, State.MISSING, "primary shard is not allocated")); } else if (!primary.started()) { - builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(primary.currentNodeId(), State.FAILED, "primary shard hasn't been started yet")); + builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(primary.currentNodeId(), State.MISSING, "primary shard hasn't been started yet")); } else { builder.put(shardId, new SnapshotMetaData.ShardSnapshotStatus(primary.currentNodeId())); } @@ -985,6 +1032,8 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL private IndicesOptions indicesOptions = IndicesOptions.strict(); + private boolean partial; + private Settings settings; private boolean includeGlobalState; @@ -1059,6 +1108,17 @@ public class SnapshotsService extends AbstractComponent implements ClusterStateL return this; } + /** + * Set to true if partial snapshot should be allowed + * + * @param partial true if partial snapshots should be allowed + * @return this request + */ + public SnapshotRequest partial(boolean partial) { + this.partial = partial; + return this; + } + /** * Returns cause for snapshot operation * diff --git a/src/test/java/org/elasticsearch/snapshots/AbstractSnapshotTests.java b/src/test/java/org/elasticsearch/snapshots/AbstractSnapshotTests.java index 9221a498d84..9114e8cc4e6 100644 --- a/src/test/java/org/elasticsearch/snapshots/AbstractSnapshotTests.java +++ b/src/test/java/org/elasticsearch/snapshots/AbstractSnapshotTests.java @@ -26,8 +26,6 @@ import org.elasticsearch.repositories.RepositoriesService; import org.elasticsearch.repositories.RepositoryMissingException; import org.elasticsearch.snapshots.mockstore.MockRepository; import org.elasticsearch.test.ElasticsearchIntegrationTest; -import org.junit.After; -import org.junit.Before; import org.junit.Ignore; import java.io.File; @@ -39,34 +37,6 @@ import static org.hamcrest.Matchers.equalTo; @Ignore public abstract class AbstractSnapshotTests extends ElasticsearchIntegrationTest { - - @After - public final void wipeAfter() { - wipeRepositories(); - } - - @Before - public final void wipeBefore() { - wipeRepositories(); - } - - /** - * Deletes repositories, supports wildcard notation. - */ - public static void wipeRepositories(String... repositories) { - // if nothing is provided, delete all - if (repositories.length == 0) { - repositories = new String[]{"*"}; - } - for (String repository : repositories) { - try { - client().admin().cluster().prepareDeleteRepository(repository).execute().actionGet(); - } catch (RepositoryMissingException ex) { - // ignore - } - } - } - public static long getFailureCount(String repository) { long failureCount = 0; for (RepositoriesService repositoriesService : cluster().getInstances(RepositoriesService.class)) { diff --git a/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java b/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java index 9e5cefa07be..c32bf948f43 100644 --- a/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java +++ b/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java @@ -22,11 +22,13 @@ package org.elasticsearch.snapshots; import com.carrotsearch.randomizedtesting.LifecycleScope; import org.elasticsearch.action.admin.cluster.repositories.put.PutRepositoryResponse; import org.elasticsearch.action.admin.cluster.snapshots.create.CreateSnapshotResponse; +import org.elasticsearch.action.admin.cluster.snapshots.restore.RestoreSnapshotResponse; import org.elasticsearch.client.Client; import org.elasticsearch.common.settings.ImmutableSettings; import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.snapshots.mockstore.MockRepositoryModule; import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.elasticsearch.test.junit.annotations.TestLogging; import org.elasticsearch.test.store.MockDirectoryHelper; import org.elasticsearch.threadpool.ThreadPool; import org.junit.Test; @@ -36,7 +38,8 @@ import java.util.ArrayList; import static com.google.common.collect.Lists.newArrayList; import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilder; import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; -import static org.hamcrest.Matchers.equalTo; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertThrows; +import static org.hamcrest.Matchers.*; /** */ @@ -126,4 +129,75 @@ public class DedicatedClusterSnapshotRestoreTests extends AbstractSnapshotTests logger.info("Number of failed shards [{}]", snapshotInfo.shardFailures().size()); logger.info("--> done"); } + + @Test + @TestLogging("snapshots:TRACE") + public void restoreIndexWithMissingShards() throws Exception { + logger.info("--> start 2 nodes"); + cluster().startNode(settingsBuilder().put("gateway.type", "local")); + cluster().startNode(settingsBuilder().put("gateway.type", "local")); + wipeIndices("_all"); + + assertAcked(prepareCreate("test-idx-1", 2, settingsBuilder().put("number_of_shards", 6) + .put("number_of_replicas", 0) + .put(MockDirectoryHelper.RANDOM_NO_DELETE_OPEN_FILE, false))); + ensureGreen(); + + logger.info("--> indexing some data into test-idx-1"); + for (int i = 0; i < 100; i++) { + index("test-idx-1", "doc", Integer.toString(i), "foo", "bar" + i); + } + refresh(); + assertThat(client().prepareCount("test-idx-1").get().getCount(), equalTo(100L)); + + logger.info("--> shutdown one of the nodes"); + cluster().stopRandomNode(); + + assertAcked(prepareCreate("test-idx-2", 1, settingsBuilder().put("number_of_shards", 6) + .put("number_of_replicas", 0) + .put(MockDirectoryHelper.RANDOM_NO_DELETE_OPEN_FILE, false))); + ensureGreen("test-idx-2"); + + logger.info("--> indexing some data into test-idx-2"); + for (int i = 0; i < 100; i++) { + index("test-idx-2", "doc", Integer.toString(i), "foo", "bar" + i); + } + refresh(); + assertThat(client().prepareCount("test-idx-2").get().getCount(), equalTo(100L)); + + logger.info("--> create repository"); + logger.info("--> creating repository"); + PutRepositoryResponse putRepositoryResponse = client().admin().cluster().preparePutRepository("test-repo") + .setType("fs").setSettings(ImmutableSettings.settingsBuilder().put("location", newTempDir())).execute().actionGet(); + assertThat(putRepositoryResponse.isAcknowledged(), equalTo(true)); + + logger.info("--> start snapshot with default settings - should fail"); + CreateSnapshotResponse createSnapshotResponse = client().admin().cluster().prepareCreateSnapshot("test-repo", "test-snap-1").setWaitForCompletion(true).execute().actionGet(); + + assertThat(createSnapshotResponse.getSnapshotInfo().state(), equalTo(SnapshotState.FAILED)); + + createSnapshotResponse = client().admin().cluster().prepareCreateSnapshot("test-repo", "test-snap-2").setWaitForCompletion(true).setPartial(true).execute().actionGet(); + logger.info("State: [{}], Reason: [{}]", createSnapshotResponse.getSnapshotInfo().state(), createSnapshotResponse.getSnapshotInfo().reason()); + assertThat(createSnapshotResponse.getSnapshotInfo().totalShards(), equalTo(12)); + assertThat(createSnapshotResponse.getSnapshotInfo().successfulShards(), lessThan(12)); + assertThat(createSnapshotResponse.getSnapshotInfo().successfulShards(), greaterThan(6)); + assertThat(client().admin().cluster().prepareGetSnapshots("test-repo").setSnapshots("test-snap-2").execute().actionGet().getSnapshots().get(0).state(), equalTo(SnapshotState.SUCCESS)); + + assertAcked(client().admin().indices().prepareClose("test-idx-1", "test-idx-2").execute().actionGet()); + + logger.info("--> restore incomplete snapshot - should fail"); + assertThrows(client().admin().cluster().prepareRestoreSnapshot("test-repo", "test-snap-2").setRestoreGlobalState(false).setWaitForCompletion(true).execute(), SnapshotRestoreException.class); + + logger.info("--> restore snapshot for the index that was snapshotted completely"); + RestoreSnapshotResponse restoreSnapshotResponse = client().admin().cluster().prepareRestoreSnapshot("test-repo", "test-snap-2").setRestoreGlobalState(false).setIndices("test-idx-2").setWaitForCompletion(true).execute().actionGet(); + assertThat(restoreSnapshotResponse.getRestoreInfo(), notNullValue()); + assertThat(restoreSnapshotResponse.getRestoreInfo().totalShards(), equalTo(6)); + assertThat(restoreSnapshotResponse.getRestoreInfo().successfulShards(), equalTo(6)); + assertThat(restoreSnapshotResponse.getRestoreInfo().failedShards(), equalTo(0)); + + ensureGreen("test-idx-2"); + + assertThat(client().prepareCount("test-idx-2").get().getCount(), equalTo(100L)); + + } } diff --git a/src/test/java/org/elasticsearch/snapshots/RepositoriesTests.java b/src/test/java/org/elasticsearch/snapshots/RepositoriesTests.java index 049f130187f..7471f1472f2 100644 --- a/src/test/java/org/elasticsearch/snapshots/RepositoriesTests.java +++ b/src/test/java/org/elasticsearch/snapshots/RepositoriesTests.java @@ -30,6 +30,8 @@ import org.elasticsearch.cluster.metadata.RepositoriesMetaData; import org.elasticsearch.cluster.metadata.RepositoryMetaData; import org.elasticsearch.common.settings.ImmutableSettings; import org.elasticsearch.repositories.RepositoryException; +import org.junit.After; +import org.junit.Before; import org.junit.Test; import static org.hamcrest.Matchers.equalTo; diff --git a/src/test/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreTests.java b/src/test/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreTests.java index 361ab50f4e5..b52d761081f 100644 --- a/src/test/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreTests.java +++ b/src/test/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreTests.java @@ -42,6 +42,8 @@ import org.elasticsearch.indices.InvalidIndexNameException; import org.elasticsearch.snapshots.mockstore.MockRepositoryModule; import org.elasticsearch.test.junit.annotations.TestLogging; import org.elasticsearch.test.store.MockDirectoryHelper; +import org.junit.After; +import org.junit.Before; import org.junit.Test; import java.io.File; @@ -53,7 +55,6 @@ import static org.hamcrest.Matchers.*; */ public class SharedClusterSnapshotRestoreTests extends AbstractSnapshotTests { - @Override public Settings indexSettings() { // During restore we frequently restore index to exactly the same state it was before, that might cause the same @@ -496,10 +497,10 @@ public class SharedClusterSnapshotRestoreTests extends AbstractSnapshotTests { logger.info("--> snapshot"); CreateSnapshotResponse createSnapshotResponse = client.admin().cluster().prepareCreateSnapshot("test-repo", "test-snap").setWaitForCompletion(true).setIndices("test-idx").get(); + assertThat(createSnapshotResponse.getSnapshotInfo().state(), equalTo(SnapshotState.FAILED)); assertThat(createSnapshotResponse.getSnapshotInfo().successfulShards(), equalTo(0)); - assertThat(createSnapshotResponse.getSnapshotInfo().totalShards(), equalTo(3)); - assertThat(createSnapshotResponse.getSnapshotInfo().shardFailures().size(), equalTo(3)); - assertThat(createSnapshotResponse.getSnapshotInfo().shardFailures().get(0).reason(), startsWith("primary shard is not allocated")); + assertThat(createSnapshotResponse.getSnapshotInfo().totalShards(), equalTo(0)); + assertThat(createSnapshotResponse.getSnapshotInfo().reason(), startsWith("Indices don't have primary shards")); } @Test diff --git a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java index f50345e1c73..32dcfd890d7 100644 --- a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java +++ b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java @@ -57,6 +57,7 @@ import org.elasticsearch.index.mapper.FieldMapper.Loading; import org.elasticsearch.index.merge.policy.*; import org.elasticsearch.indices.IndexMissingException; import org.elasticsearch.indices.IndexTemplateMissingException; +import org.elasticsearch.repositories.RepositoryMissingException; import org.elasticsearch.rest.RestStatus; import org.elasticsearch.search.SearchService; import org.junit.After; @@ -189,6 +190,7 @@ public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase wipeIndices("_all"); wipeTemplates(); randomIndexTemplate(); + wipeRepositories(); logger.info("[{}#{}]: before test", getTestClass().getSimpleName(), getTestName()); } catch (OutOfMemoryError e) { if (e.getMessage().contains("unable to create new native thread")) { @@ -237,6 +239,7 @@ public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase wipeIndices("_all"); // wipe after to make sure we fail in the test that // didn't ack the delete wipeTemplates(); + wipeRepositories(); ensureAllSearchersClosed(); ensureAllFilesClosed(); logger.info("[{}#{}]: cleaned up after test", getTestClass().getSimpleName(), getTestName()); @@ -368,6 +371,25 @@ public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase } } + /** + * Deletes repositories, supports wildcard notation. + */ + public static void wipeRepositories(String... repositories) { + if (cluster().size() > 0) { + // if nothing is provided, delete all + if (repositories.length == 0) { + repositories = new String[]{"*"}; + } + for (String repository : repositories) { + try { + client().admin().cluster().prepareDeleteRepository(repository).execute().actionGet(); + } catch (RepositoryMissingException ex) { + // ignore + } + } + } + } + /** * Creates one or more indices and asserts that the indices are acknowledged. If one of the indices * already exists this method will fail and wipe all the indices created so far. From 22a96e6a18819efc17076a0f0c6a79d9e2b2a902 Mon Sep 17 00:00:00 2001 From: Benjamin Vetter Date: Sun, 12 Jan 2014 11:26:39 +0100 Subject: [PATCH 06/34] Added stopwords: _none_ to the docs #329 --- docs/reference/analysis/analyzers.asciidoc | 8 ++++++++ .../analysis/analyzers/standard-analyzer.asciidoc | 2 +- docs/reference/analysis/analyzers/stop-analyzer.asciidoc | 2 +- 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/reference/analysis/analyzers.asciidoc b/docs/reference/analysis/analyzers.asciidoc index b97231b0ab3..22881a68904 100644 --- a/docs/reference/analysis/analyzers.asciidoc +++ b/docs/reference/analysis/analyzers.asciidoc @@ -49,6 +49,14 @@ index : stopwords : [test1, test2, test3] -------------------------------------------------- +[[analyzers-stopwords]] +=== Stopwords + +The `stopwords` parameter can be used to provide a custom set of stopwords. As +certain analyzers use a default list of stopwords while others don't, please +check out the individual analyzer sections. In case you want the analyzer to +use no stopwords at all, simply provide `stopwords: _none_` + Below is a list of the built in analyzers. include::analyzers/standard-analyzer.asciidoc[] diff --git a/docs/reference/analysis/analyzers/standard-analyzer.asciidoc b/docs/reference/analysis/analyzers/standard-analyzer.asciidoc index 14027ffde0b..0b6faed7712 100644 --- a/docs/reference/analysis/analyzers/standard-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/standard-analyzer.asciidoc @@ -17,7 +17,7 @@ type: [cols="<,<",options="header",] |======================================================================= |Setting |Description -|`stopwords` |A list of stopword to initialize the stop filter with. +|`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to an 'empty' stopword list added[1.0.0.Beta1, Previously defaulted to the English stopwords list] |`max_token_length` |The maximum token length. If a token is seen that diff --git a/docs/reference/analysis/analyzers/stop-analyzer.asciidoc b/docs/reference/analysis/analyzers/stop-analyzer.asciidoc index 18599af0f90..2a1bfd73113 100644 --- a/docs/reference/analysis/analyzers/stop-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/stop-analyzer.asciidoc @@ -12,7 +12,7 @@ The following are settings that can be set for a `stop` analyzer type: [cols="<,<",options="header",] |======================================================================= |Setting |Description -|`stopwords` |A list of stopword to initialize the stop filter with. +|`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to the english stop words. |`stopwords_path` |A path (either relative to `config` location, or From ba8e012be90c8f3847bb638a99a75a1b994cf8c5 Mon Sep 17 00:00:00 2001 From: Benjamin Vetter Date: Mon, 13 Jan 2014 20:04:30 +0100 Subject: [PATCH 07/34] Referring to stop analyzer for stopword docs #329 --- docs/reference/analysis/analyzers.asciidoc | 8 -------- docs/reference/analysis/analyzers/lang-analyzer.asciidoc | 3 ++- .../analysis/analyzers/pattern-analyzer.asciidoc | 3 ++- .../analysis/analyzers/snowball-analyzer.asciidoc | 5 +++-- .../analysis/analyzers/standard-analyzer.asciidoc | 3 ++- docs/reference/analysis/analyzers/stop-analyzer.asciidoc | 3 ++- 6 files changed, 11 insertions(+), 14 deletions(-) diff --git a/docs/reference/analysis/analyzers.asciidoc b/docs/reference/analysis/analyzers.asciidoc index 22881a68904..b97231b0ab3 100644 --- a/docs/reference/analysis/analyzers.asciidoc +++ b/docs/reference/analysis/analyzers.asciidoc @@ -49,14 +49,6 @@ index : stopwords : [test1, test2, test3] -------------------------------------------------- -[[analyzers-stopwords]] -=== Stopwords - -The `stopwords` parameter can be used to provide a custom set of stopwords. As -certain analyzers use a default list of stopwords while others don't, please -check out the individual analyzer sections. In case you want the analyzer to -use no stopwords at all, simply provide `stopwords: _none_` - Below is a list of the built in analyzers. include::analyzers/standard-analyzer.asciidoc[] diff --git a/docs/reference/analysis/analyzers/lang-analyzer.asciidoc b/docs/reference/analysis/analyzers/lang-analyzer.asciidoc index d3505dd1d37..f963e4b0f1b 100644 --- a/docs/reference/analysis/analyzers/lang-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/lang-analyzer.asciidoc @@ -11,7 +11,8 @@ following types are supported: `arabic`, `armenian`, `basque`, All analyzers support setting custom `stopwords` either internally in the config, or by using an external stopwords file by setting -`stopwords_path`. +`stopwords_path`. Check <> for +more details. The following analyzers support setting custom `stem_exclusion` list: `arabic`, `armenian`, `basque`, `brazilian`, `bulgarian`, `catalan`, diff --git a/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc b/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc index 04f71bfc773..12f8799f8f9 100644 --- a/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc @@ -15,7 +15,8 @@ type: |`flags` |The regular expression flags. |`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to an 'empty' stopword list coming[1.0.0.RC1, Previously -defaulted to the English stopwords list] +defaulted to the English stopwords list]. Check +<> for more details. |=================================================================== *IMPORTANT*: The regular expression should match the *token separators*, diff --git a/docs/reference/analysis/analyzers/snowball-analyzer.asciidoc b/docs/reference/analysis/analyzers/snowball-analyzer.asciidoc index 234f41db155..64804fcb359 100644 --- a/docs/reference/analysis/analyzers/snowball-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/snowball-analyzer.asciidoc @@ -41,8 +41,9 @@ filter>> and defaults to `English`. Note that not all the language analyzers have a default set of stopwords provided. The `stopwords` parameter can be used to provide stopwords for the -languages that has no defaults, or to simply replace the default set -with your custom list. A default set of stopwords for many of these +languages that have no defaults, or to simply replace the default set +with your custom list. Check <> +for more details. A default set of stopwords for many of these languages is available from for instance https://github.com/apache/lucene-solr/tree/trunk/lucene/analysis/common/src/resources/org/apache/lucene/analysis/[here] and diff --git a/docs/reference/analysis/analyzers/standard-analyzer.asciidoc b/docs/reference/analysis/analyzers/standard-analyzer.asciidoc index 0b6faed7712..4aae94a69bb 100644 --- a/docs/reference/analysis/analyzers/standard-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/standard-analyzer.asciidoc @@ -19,7 +19,8 @@ type: |Setting |Description |`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to an 'empty' stopword list added[1.0.0.Beta1, Previously -defaulted to the English stopwords list] +defaulted to the English stopwords list]. Check +<> for more details. |`max_token_length` |The maximum token length. If a token is seen that exceeds this length then it is discarded. Defaults to `255`. |======================================================================= diff --git a/docs/reference/analysis/analyzers/stop-analyzer.asciidoc b/docs/reference/analysis/analyzers/stop-analyzer.asciidoc index 2a1bfd73113..9a19772795f 100644 --- a/docs/reference/analysis/analyzers/stop-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/stop-analyzer.asciidoc @@ -14,8 +14,9 @@ The following are settings that can be set for a `stop` analyzer type: |Setting |Description |`stopwords` |A list of stopwords to initialize the stop filter with. Defaults to the english stop words. - |`stopwords_path` |A path (either relative to `config` location, or absolute) to a stopwords file configuration. |======================================================================= +Use `stopwords: _none_` to explicitly specify an 'empty' stopword list. + From 6829b67ff02602f0250b56df4b0e65b4689d6368 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 14 Jan 2014 16:11:45 +0100 Subject: [PATCH 08/34] Fix Release tool to run smoke tests off tags this commit allows to run the release tool for smoke testing without being on the actually released branch. This commit also added a list of plugins that will be installed for smoke testing to see if the plugin startup mechanism works correctly. --- dev-tools/build_release.py | 78 +++++++++++++++++++++++++++++--------- 1 file changed, 60 insertions(+), 18 deletions(-) diff --git a/dev-tools/build_release.py b/dev-tools/build_release.py index ed6a5a0e657..a49b8d0e445 100644 --- a/dev-tools/build_release.py +++ b/dev-tools/build_release.py @@ -58,6 +58,12 @@ Once it's done it will print all the remaining steps. """ env = os.environ +PLUGINS = [('bigdesk', 'lukas-vlcek/bigdesk'), + ('paramedic', 'karmi/elasticsearch-paramedic'), + ('segmentspy', 'polyfractal/elasticsearch-segmentspy'), + ('inquisitor', 'polyfractal/elasticsearch-inquisitor'), + ('head', 'mobz/elasticsearch-head')] + LOG = env.get('ES_RELEASE_LOG', '/tmp/elasticsearch_release.log') def log(msg): @@ -117,10 +123,11 @@ def verify_mvn_java_version(version, mvn): # Returns the hash of the current git HEAD revision def get_head_hash(): - return get_hash('HEAD') + return os.popen(' git rev-parse --verify HEAD 2>&1').read().strip() -def get_hash(version): - return os.popen('git rev-parse --verify %s 2>&1' % (version)).read().strip() +# Returns the hash of the given tag revision +def get_tag_hash(tag): + return os.popen('git show-ref --tags %s --hash 2>&1' % (tag)).read().strip() # Returns the name of the current branch def get_current_branch(): @@ -133,6 +140,10 @@ verify_mvn_java_version('1.6', MVN) def release_branch(version): return 'release_branch_%s' % version +# runs get fetch on the given remote +def fetch(remote): + run('git fetch %s' % remote) + # Creates a new release branch from the given source branch # and rebases the source branch from the remote before creating # the release branch. Note: This fails if the source branch @@ -309,7 +320,7 @@ def generate_checksums(files): res = res + [os.path.join(directory, checksum_file), release_file] return res -def download_and_verify(release, files, base_url='https://download.elasticsearch.org/elasticsearch/elasticsearch'): +def download_and_verify(release, files, plugins=None, base_url='https://download.elasticsearch.org/elasticsearch/elasticsearch'): print('Downloading and verifying release %s from %s' % (release, base_url)) tmp_dir = tempfile.mkdtemp() try: @@ -326,11 +337,11 @@ def download_and_verify(release, files, base_url='https://download.elasticsearch urllib.request.urlretrieve(url, checksum_file) print(' Verifying checksum %s' % (checksum_file)) run('cd %s && sha1sum -c %s' % (tmp_dir, os.path.basename(checksum_file))) - smoke_test_release(release, downloaded_files, get_hash('v%s' % release)) + smoke_test_release(release, downloaded_files, get_tag_hash('v%s' % release), plugins) finally: shutil.rmtree(tmp_dir) -def smoke_test_release(release, files, expected_hash): +def smoke_test_release(release, files, expected_hash, plugins): for release_file in files: if not os.path.isfile(release_file): raise RuntimeError('Smoketest failed missing file %s' % (release_file)) @@ -344,9 +355,20 @@ def smoke_test_release(release, files, expected_hash): continue # nothing to do here es_run_path = os.path.join(tmp_dir, 'elasticsearch-%s' % (release), 'bin/elasticsearch') print(' Smoke testing package [%s]' % release_file) + es_plugin_path = os.path.join(tmp_dir, 'elasticsearch-%s' % (release),'bin/plugin') + plugin_names = {} + for name, plugin in plugins: + print(' Install plugin [%s] from [%s]' % (name, plugin)) + run('%s %s %s' % (es_plugin_path, '-install', plugin)) + plugin_names[name] = True + + if release.startswith("0.90."): + background = '' # 0.90.x starts in background automatically + else: + background = '-d' print(' Starting elasticsearch deamon from [%s]' % os.path.join(tmp_dir, 'elasticsearch-%s' % release)) - run('%s; %s -Des.node.name=smoke_tester -Des.cluster.name=prepare_release -Des.discovery.zen.ping.multicast.enabled=false -d' - % (java_exe(), es_run_path)) + run('%s; %s -Des.node.name=smoke_tester -Des.cluster.name=prepare_release -Des.discovery.zen.ping.multicast.enabled=false %s' + % (java_exe(), es_run_path, background)) conn = HTTPConnection('127.0.0.1', 9200, 20); wait_for_node_startup() try: @@ -360,9 +382,25 @@ def smoke_test_release(release, files, expected_hash): if version['build_snapshot']: raise RuntimeError('Expected non snapshot version') if version['build_hash'].strip() != expected_hash: - raise RuntimeError('HEAD hash does not match expected [%s] but got [%s]' % (get_head_hash(), version['build_hash'])) + raise RuntimeError('HEAD hash does not match expected [%s] but got [%s]' % (expected_hash, version['build_hash'])) print(' Running REST Spec tests against package [%s]' % release_file) run_mvn('test -Dtests.rest=%s -Dtests.class=*.*RestTests' % ("127.0.0.1:9200")) + print(' Verify if plugins are listed in _nodes') + conn.request('GET', '/_nodes?plugin=true&pretty=true') + res = conn.getresponse() + if res.status == 200: + nodes = json.loads(res.read().decode("utf-8"))['nodes'] + for _, node in nodes.items(): + node_plugins = node['plugins'] + for node_plugin in node_plugins: + if not plugin_names.get(node_plugin['name'], False): + raise RuntimeError('Unexpeced plugin %s' % node_plugin['name']) + del plugin_names[node_plugin['name']] + if plugin_names: + raise RuntimeError('Plugins not loaded %s' % list(plugin_names.keys())) + + else: + raise RuntimeError('Expected HTTP 200 but got %s' % res.status) else: raise RuntimeError('Expected HTTP 200 but got %s' % res.status) finally: @@ -471,14 +509,11 @@ if __name__ == '__main__': print('Preparing Release from branch [%s] running tests: [%s] dryrun: [%s]' % (src_branch, run_tests, dry_run)) print(' JAVA_HOME is [%s]' % JAVA_HOME) print(' Running with maven command: [%s] ' % (MVN)) - release_version = find_release_version(src_branch) - - if not smoke_test_version and not dry_run: - smoke_test_version = release_version - elif smoke_test_version: - print("Skipping build - smoketest only against version %s" % smoke_test_version) if build: + release_version = find_release_version(src_branch) + if not dry_run: + smoke_test_version = release_version head_hash = get_head_hash() run_mvn('clean') # clean the env! print(' Release version: [%s]' % release_version) @@ -497,11 +532,14 @@ if __name__ == '__main__': print(''.join(['-' for _ in range(80)])) print('Building Release candidate') input('Press Enter to continue...') - print(' Running maven builds now and publish to sonartype- run-tests [%s]' % run_tests) + if not dry_run: + print(' Running maven builds now and publish to sonartype - run-tests [%s]' % run_tests) + else: + print(' Running maven builds now run-tests [%s]' % run_tests) build_release(run_tests=run_tests, dry_run=dry_run, cpus=cpus) artifacts = get_artifacts(release_version) artifacts_and_checksum = generate_checksums(artifacts) - smoke_test_release(release_version, artifacts, get_head_hash()) + smoke_test_release(release_version, artifacts, get_head_hash(), PLUGINS) print(''.join(['-' for _ in range(80)])) print('Finish Release -- dry_run: %s' % dry_run) input('Press Enter to continue...') @@ -530,5 +568,9 @@ if __name__ == '__main__': run('git tag -d v%s' % release_version) # we delete this one anyways run('git branch -D %s' % (release_branch(release_version))) + else: + print("Skipping build - smoketest only against version %s" % smoke_test_version) + if smoke_test_version: - download_and_verify(smoke_test_version, artifact_names(smoke_test_version)) + fetch(remote) + download_and_verify(smoke_test_version, artifact_names(smoke_test_version), plugins=PLUGINS) From 99eae50f9c23bfe73a7a52273dab06fb196b1e10 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 14 Jan 2014 17:00:32 +0100 Subject: [PATCH 09/34] run mvn clean before smoketesting --- dev-tools/build_release.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dev-tools/build_release.py b/dev-tools/build_release.py index a49b8d0e445..6566b34d576 100644 --- a/dev-tools/build_release.py +++ b/dev-tools/build_release.py @@ -338,6 +338,7 @@ def download_and_verify(release, files, plugins=None, base_url='https://download print(' Verifying checksum %s' % (checksum_file)) run('cd %s && sha1sum -c %s' % (tmp_dir, os.path.basename(checksum_file))) smoke_test_release(release, downloaded_files, get_tag_hash('v%s' % release), plugins) + print(' SUCCESS') finally: shutil.rmtree(tmp_dir) @@ -570,6 +571,7 @@ if __name__ == '__main__': run('git branch -D %s' % (release_branch(release_version))) else: print("Skipping build - smoketest only against version %s" % smoke_test_version) + run_mvn('clean') # clean the env! if smoke_test_version: fetch(remote) From f2710c16ebd918f646be9d0ab64b4871c25be4c2 Mon Sep 17 00:00:00 2001 From: Rob Cherry Date: Mon, 4 Nov 2013 12:55:27 -0500 Subject: [PATCH 10/34] excluding all fields of an object should not remove parent. When excluding '*.f1' from `{ "obj": { "f1": 1, "f2": 2 } }` XContentMapValues.filter returns `{ "obj": { "f2": 2}}`. When run on `{ "obj": { "f1" : 1 }}` we should return `{ "obj": { }}` to maintain object structure. People currently need to always check whether `obj` is there or not. Closes #4715 Closes #4047 Related to #4491 --- .../xcontent/support/XContentMapValues.java | 29 ++--- .../support/XContentMapValuesTests.java | 117 +++++++++++++++--- .../search/fields/SearchFieldsTests.java | 2 +- 3 files changed, 114 insertions(+), 34 deletions(-) diff --git a/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java b/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java index 27b2c1ba67e..ab316e9f31a 100644 --- a/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java +++ b/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java @@ -154,24 +154,18 @@ public class XContentMapValues { } sb.append(key); String path = sb.toString(); - boolean excluded = false; - for (String exclude : excludes) { - if (Regex.simpleMatch(exclude, path)) { - excluded = true; - break; - } - } - if (excluded) { + + if (Regex.simpleMatch(excludes, path)) { sb.setLength(mark); continue; } - boolean exactIncludeMatch; + + boolean exactIncludeMatch = false; // true if the current position was specifically mentioned + boolean pathIsPrefixOfAnInclude = false; // true if potentially a sub scope can be included if (includes.length == 0) { // implied match anything exactIncludeMatch = true; } else { - exactIncludeMatch = false; - boolean pathIsPrefixOfAnInclude = false; for (String include : includes) { // check for prefix matches as well to see if we need to zero in, something like: obj1.arr1.* or *.field // note, this does not work well with middle matches, like obj1.*.obj3 @@ -198,11 +192,12 @@ public class XContentMapValues { break; } } - if (!pathIsPrefixOfAnInclude && !exactIncludeMatch) { - // skip subkeys, not interesting. - sb.setLength(mark); - continue; - } + } + + if (!(pathIsPrefixOfAnInclude || exactIncludeMatch)) { + // skip subkeys, not interesting. + sb.setLength(mark); + continue; } @@ -210,7 +205,7 @@ public class XContentMapValues { Map innerInto = Maps.newHashMap(); // if we had an exact match, we want give deeper excludes their chance filter((Map) entry.getValue(), innerInto, exactIncludeMatch ? Strings.EMPTY_ARRAY : includes, excludes, sb); - if (!innerInto.isEmpty()) { + if (exactIncludeMatch || !innerInto.isEmpty()) { into.put(entry.getKey(), innerInto); } } else if (entry.getValue() instanceof List) { diff --git a/src/test/java/org/elasticsearch/common/xcontent/support/XContentMapValuesTests.java b/src/test/java/org/elasticsearch/common/xcontent/support/XContentMapValuesTests.java index eff2bdd4506..e2fba5ac170 100644 --- a/src/test/java/org/elasticsearch/common/xcontent/support/XContentMapValuesTests.java +++ b/src/test/java/org/elasticsearch/common/xcontent/support/XContentMapValuesTests.java @@ -26,6 +26,7 @@ import org.elasticsearch.common.xcontent.XContentFactory; import org.elasticsearch.common.xcontent.XContentHelper; import org.elasticsearch.common.xcontent.XContentType; import org.elasticsearch.test.ElasticsearchTestCase; +import org.hamcrest.Matchers; import org.junit.Test; import java.util.Arrays; @@ -33,7 +34,6 @@ import java.util.HashMap; import java.util.List; import java.util.Map; -import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.*; import static org.hamcrest.core.IsEqual.equalTo; @@ -46,6 +46,7 @@ public class XContentMapValuesTests extends ElasticsearchTestCase { XContentBuilder builder = XContentFactory.jsonBuilder().startObject() .field("test1", "value1") .field("test2", "value2") + .field("something_else", "value3") .endObject(); Map source = XContentFactory.xContent(XContentType.JSON).createParser(builder.string()).mapAndClose(); @@ -59,8 +60,9 @@ public class XContentMapValuesTests extends ElasticsearchTestCase { assertThat(filter.get("test2").toString(), equalTo("value2")); filter = XContentMapValues.filter(source, Strings.EMPTY_ARRAY, new String[]{"test1"}); - assertThat(filter.size(), equalTo(1)); + assertThat(filter.size(), equalTo(2)); assertThat(filter.get("test2").toString(), equalTo("value2")); + assertThat(filter.get("something_else").toString(), equalTo("value3")); // more complex object... builder = XContentFactory.jsonBuilder().startObject() @@ -200,20 +202,6 @@ public class XContentMapValuesTests extends ElasticsearchTestCase { assertThat(XContentMapValues.extractRawValues("path1.xxx.path2.yyy.test", map).get(0).toString(), equalTo("value")); } - @Test - public void testThatFilteringWithNestedArrayAndExclusionWorks() throws Exception { - XContentBuilder builder = XContentFactory.jsonBuilder().startObject() - .startArray("coordinates") - .startArray().value("foo").endArray() - .endArray() - .endObject(); - - Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); - Map filteredSource = XContentMapValues.filter(mapTuple.v2(), Strings.EMPTY_ARRAY, new String[]{"nonExistingField"}); - - assertThat(mapTuple.v2(), equalTo(filteredSource)); - } - @Test public void prefixedNamesFilteringTest() { Map map = new HashMap(); @@ -368,4 +356,101 @@ public class XContentMapValuesTests extends ElasticsearchTestCase { assertThat(filteredMap.get("field").toString(), equalTo("value")); } + + @SuppressWarnings({"unchecked"}) + @Test + public void testThatFilterIncludesEmptyObjectWhenUsingIncludes() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj") + .endObject() + .endObject(); + + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), new String[]{"obj"}, Strings.EMPTY_ARRAY); + + assertThat(mapTuple.v2(), equalTo(filteredSource)); + } + + @Test + public void testThatFilterIncludesEmptyObjectWhenUsingExcludes() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj") + .endObject() + .endObject(); + + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), Strings.EMPTY_ARRAY, new String[]{"nonExistingField"}); + + assertThat(mapTuple.v2(), equalTo(filteredSource)); + } + + @Test + public void testNotOmittingObjectsWithExcludedProperties() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj") + .field("f1", "v1") + .endObject() + .endObject(); + + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), Strings.EMPTY_ARRAY, new String[]{"obj.f1"}); + + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj")); + assertThat(((Map) filteredSource.get("obj")).size(), equalTo(0)); + } + + @SuppressWarnings({"unchecked"}) + @Test + public void testNotOmittingObjectWithNestedExcludedObject() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj1") + .startObject("obj2") + .startObject("obj3") + .endObject() + .endObject() + .endObject() + .endObject(); + + // implicit include + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), Strings.EMPTY_ARRAY, new String[]{"*.obj2"}); + + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj1")); + assertThat(((Map) filteredSource.get("obj1")).size(), Matchers.equalTo(0)); + + // explicit include + filteredSource = XContentMapValues.filter(mapTuple.v2(), new String[]{"obj1"}, new String[]{"*.obj2"}); + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj1")); + assertThat(((Map) filteredSource.get("obj1")).size(), Matchers.equalTo(0)); + + // wild card include + filteredSource = XContentMapValues.filter(mapTuple.v2(), new String[]{"*.obj2"}, new String[]{"*.obj3"}); + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj1")); + assertThat(((Map) filteredSource.get("obj1")), hasKey("obj2")); + assertThat(((Map) ((Map) filteredSource.get("obj1")).get("obj2")).size(), Matchers.equalTo(0)); + } + + @SuppressWarnings({"unchecked"}) + @Test + public void testIncludingObjectWithNestedIncludedObject() throws Exception { + XContentBuilder builder = XContentFactory.jsonBuilder().startObject() + .startObject("obj1") + .startObject("obj2") + .endObject() + .endObject() + .endObject(); + + Tuple> mapTuple = XContentHelper.convertToMap(builder.bytes(), true); + Map filteredSource = XContentMapValues.filter(mapTuple.v2(), new String[]{"*.obj2"}, Strings.EMPTY_ARRAY); + + assertThat(filteredSource.size(), equalTo(1)); + assertThat(filteredSource, hasKey("obj1")); + assertThat(((Map) filteredSource.get("obj1")).size(), equalTo(1)); + assertThat(((Map) filteredSource.get("obj1")), hasKey("obj2")); + assertThat(((Map) ((Map) filteredSource.get("obj1")).get("obj2")).size(), equalTo(0)); + } } diff --git a/src/test/java/org/elasticsearch/search/fields/SearchFieldsTests.java b/src/test/java/org/elasticsearch/search/fields/SearchFieldsTests.java index 4abed6dbb72..314c235cf38 100644 --- a/src/test/java/org/elasticsearch/search/fields/SearchFieldsTests.java +++ b/src/test/java/org/elasticsearch/search/fields/SearchFieldsTests.java @@ -253,7 +253,7 @@ public class SearchFieldsTests extends ElasticsearchIntegrationTest { SearchResponse response = client().prepareSearch("test") .addPartialField("partial1", "obj1.arr1.*", null) - .addPartialField("partial2", null, "obj1.*") + .addPartialField("partial2", null, "obj1") .execute().actionGet(); assertThat("Failures " + Arrays.toString(response.getShardFailures()), response.getShardFailures().length, equalTo(0)); From 9440655ab9091cefe33fe1853f91a0897670fc55 Mon Sep 17 00:00:00 2001 From: Igor Motov Date: Tue, 14 Jan 2014 10:43:15 -0500 Subject: [PATCH 11/34] Fix possible race condition in the restoreIndexWithMissingShards test Due to a race condition the index creation operation might still try to create an index on the closing node. --- .../snapshots/DedicatedClusterSnapshotRestoreTests.java | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java b/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java index c32bf948f43..7728007d67b 100644 --- a/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java +++ b/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java @@ -24,6 +24,7 @@ import org.elasticsearch.action.admin.cluster.repositories.put.PutRepositoryResp import org.elasticsearch.action.admin.cluster.snapshots.create.CreateSnapshotResponse; import org.elasticsearch.action.admin.cluster.snapshots.restore.RestoreSnapshotResponse; import org.elasticsearch.client.Client; +import org.elasticsearch.common.Priority; import org.elasticsearch.common.settings.ImmutableSettings; import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.snapshots.mockstore.MockRepositoryModule; @@ -152,6 +153,7 @@ public class DedicatedClusterSnapshotRestoreTests extends AbstractSnapshotTests logger.info("--> shutdown one of the nodes"); cluster().stopRandomNode(); + assertThat(client().admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setTimeout("1m").setWaitForNodes("<2").execute().actionGet().isTimedOut(), equalTo(false)); assertAcked(prepareCreate("test-idx-2", 1, settingsBuilder().put("number_of_shards", 6) .put("number_of_replicas", 0) From d1a4f889aeabe1a99cd0ba62766b585715f1623b Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 14 Jan 2014 19:38:39 +0100 Subject: [PATCH 12/34] Fix DestructiveOperationsIntegrationTests to wait for index to be allocated before closing --- ...DestructiveOperationsIntegrationTests.java | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java b/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java index b7a26dd185a..cd0ab6efec8 100644 --- a/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java +++ b/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java @@ -53,12 +53,12 @@ public class DestructiveOperationsIntegrationTests extends ElasticsearchIntegrat try { // should fail since index1 is the only index. client().admin().indices().prepareDelete("i*").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { client().admin().indices().prepareDelete("_all").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} settings = ImmutableSettings.builder() @@ -78,26 +78,26 @@ public class DestructiveOperationsIntegrationTests extends ElasticsearchIntegrat assertAcked(client().admin().indices().prepareCreate("index1").get()); assertAcked(client().admin().indices().prepareCreate("1index").get()); - + ensureYellow();// wait for primaries to be allocated // Should succeed, since no wildcards assertAcked(client().admin().indices().prepareClose("1index").get()); try { client().admin().indices().prepareClose("_all").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { assertAcked(client().admin().indices().prepareOpen("_all").get()); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) { } try { client().admin().indices().prepareClose("*").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { assertAcked(client().admin().indices().prepareOpen("*").get()); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) { } @@ -124,12 +124,12 @@ public class DestructiveOperationsIntegrationTests extends ElasticsearchIntegrat try { client().prepareDeleteByQuery("_all").setQuery(QueryBuilders.matchAllQuery()).get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { client().prepareDeleteByQuery().setQuery(QueryBuilders.matchAllQuery()).get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} settings = ImmutableSettings.builder() @@ -157,12 +157,12 @@ public class DestructiveOperationsIntegrationTests extends ElasticsearchIntegrat client().admin().indices().prepareDeleteMapping("1index").setType("1").get(); try { client().admin().indices().prepareDeleteMapping("_all").setType("1").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} try { client().admin().indices().prepareDeleteMapping().setType("1").get(); - assert false; + fail(); } catch (ElasticsearchIllegalArgumentException e) {} settings = ImmutableSettings.builder() @@ -170,10 +170,10 @@ public class DestructiveOperationsIntegrationTests extends ElasticsearchIntegrat .build(); assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(settings)); - client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string").get(); - client().admin().indices().prepareDeleteMapping().setType("1").get(); - client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string").get(); - client().admin().indices().prepareDeleteMapping("_all").setType("1").get(); + assertAcked(client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string")); + assertAcked(client().admin().indices().prepareDeleteMapping().setType("1")); + assertAcked(client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string")); + assertAcked(client().admin().indices().prepareDeleteMapping("_all").setType("1")); } } From 411739fe3b5b2410fa9594edf27087718162225f Mon Sep 17 00:00:00 2001 From: Britta Weber Date: Wed, 8 Jan 2014 10:34:48 +0100 Subject: [PATCH 13/34] Make PUT and DELETE consistent for _mapping, _alias and _warmer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit See issue #4071 PUT options for _mapping: Single type can now be added with `[PUT|POST] {index|_all|*|regex|blank}/[_mapping|_mappings]/type` and `[PUT|POST] {index|_all|*|regex|blank}/type/[_mapping|_mappings]` PUT options for _warmer: PUT with a single warmer can now be done with `[PUT|POST] {index|_all|*|prefix*|blank}/{type|_all|*|prefix*|blank}/[_warmer|_warmers]/warmer_name` PUT options for _alias: Single alias can now be PUT with `[PUT|POST] {index|_all|*|prefix*|blank}/[_alias|_aliases]/alias` DELETE options _mapping: Several mappings can be deleted at once by defining several indices and types with `[DELETE] /{index}/{type}` `[DELETE] /{index}/{type}/_mapping` `[DELETE] /{index}/_mapping/{type}` where `index= * | _all | glob pattern | name1, name2, …` `type= * | _all | glob pattern | name1, name2, …` Alternatively, the keyword `_mapings` can be used. DELETE options for _warmer: Several warmers can be deleted at once by defining several indices and names with `[DELETE] /{index}/_warmer/{type}` where `index= * | _all | glob pattern | name1, name2, …` `type= * | _all | glob pattern | name1, name2, …` Alternatively, the keyword `_warmers` can be used. DELETE options for _alias: Several aliases can be deleted at once by defining several indices and names with `[DELETE] /{index}/_alias/{type}` where `index= * | _all | glob pattern | name1, name2, …` `type= * | _all | glob pattern | name1, name2, …` Alternatively, the keyword `_aliases` can be used. --- docs/reference/indices/aliases.asciidoc | 29 +- .../reference/indices/delete-mapping.asciidoc | 21 +- docs/reference/indices/put-mapping.asciidoc | 22 ++ docs/reference/indices/warmers.asciidoc | 56 ++- rest-api-spec/api/indices.delete_alias.json | 2 +- rest-api-spec/api/indices.delete_mapping.json | 2 +- rest-api-spec/api/indices.delete_warmer.json | 4 +- rest-api-spec/api/indices.put_alias.json | 4 +- rest-api-spec/api/indices.put_mapping.json | 4 +- rest-api-spec/api/indices.put_warmer.json | 4 +- .../all_path_options.yaml | 225 ++++++++++++ .../all_path_options.yaml | 286 ++++++++++++++++ .../all_path_options.yaml | 214 ++++++++++++ .../indices.put_alias/all_path_options.yaml | 127 +++++++ .../indices.put_mapping/all_path_options.yaml | 178 ++++++++++ .../all_path_options.yaml | 113 +++++++ .../test/indices.put_warmer/10_basic.yaml | 1 + .../indices.put_warmer/all_path_options.yaml | 128 +++++++ .../indices/alias/IndicesAliasesRequest.java | 319 +++++++++++++----- .../alias/IndicesAliasesRequestBuilder.java | 129 +++++-- .../alias/TransportIndicesAliasesAction.java | 40 ++- ...eleteMappingClusterStateUpdateRequest.java | 10 +- .../mapping/delete/DeleteMappingRequest.java | 62 ++-- .../delete/DeleteMappingRequestBuilder.java | 4 +- .../delete/TransportDeleteMappingAction.java | 28 +- .../warmer/delete/DeleteWarmerRequest.java | 49 ++- .../delete/DeleteWarmerRequestBuilder.java | 4 +- .../delete/TransportDeleteWarmerAction.java | 31 +- .../cluster/metadata/AliasAction.java | 66 ++-- .../cluster/metadata/MetaData.java | 34 +- .../metadata/MetaDataMappingService.java | 23 +- .../common/util/CollectionUtils.java | 11 + .../indices/TypeMissingException.java | 10 +- .../alias/delete/AliasesMissingException.java | 46 +++ .../delete/RestIndexDeleteAliasesAction.java | 8 +- .../alias/put/RestIndexPutAliasAction.java | 20 +- .../delete/RestDeleteMappingAction.java | 7 +- .../mapping/put/RestPutMappingAction.java | 20 +- .../warmer/delete/RestDeleteWarmerAction.java | 5 +- .../warmer/put/RestPutWarmerAction.java | 14 + .../warmer/IndexWarmerMissingException.java | 14 +- .../delete/DeleteWarmerRequestTests.java | 4 +- .../aliases/IndexAliasesTests.java | 61 +++- .../elasticsearch/cluster/ack/AckTests.java | 2 +- .../cluster/metadata/MetaDataTests.java | 12 +- .../indices/IndicesOptionsTests.java | 120 ++++++- .../mapping/SimpleDeleteMappingTests.java | 40 ++- .../LocalGatewayIndicesWarmerTests.java | 2 +- .../warmer/SimpleIndicesWarmerTests.java | 6 +- .../mlt/MoreLikeThisActionTests.java | 4 +- ...DestructiveOperationsIntegrationTests.java | 18 +- 51 files changed, 2330 insertions(+), 313 deletions(-) create mode 100644 rest-api-spec/test/indices.delete_alias/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.delete_mapping/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.delete_warmer/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.put_alias/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.put_mapping/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.put_settings/all_path_options.yaml create mode 100644 rest-api-spec/test/indices.put_warmer/all_path_options.yaml create mode 100644 src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/AliasesMissingException.java diff --git a/docs/reference/indices/aliases.asciidoc b/docs/reference/indices/aliases.asciidoc index 5451ac530b3..53f484cadcf 100644 --- a/docs/reference/indices/aliases.asciidoc +++ b/docs/reference/indices/aliases.asciidoc @@ -153,17 +153,22 @@ curl -XGET 'http://localhost:9200/alias2/_search?q=user:kimchy&routing=2,3' [float] [[alias-adding]] -=== Add a single index alias +=== Add a single alias -There is also an api to add a single index alias, with options: +An alias can also be added with the endpoint + +`PUT /{index}/_alias/{name}` + + +where [horizontal] -`index`:: The index to alias refers to. This is a required option. -`alias`:: The name of the alias. This is a required option. +`index`:: The index to alias refers to. Can be any of `blank | * | _all | glob pattern | name1, name2, …` +`name`:: The name of the alias. This is a required option. `routing`:: An optional routing that can be associated with an alias. `filter`:: An optional filter that can be associated with an alias. -The rest endpoint is: `/{index}/_alias/{alias}`. +You can also use the plural `_aliases`. [float] ==== Examples: @@ -191,16 +196,18 @@ curl -XPUT 'localhost:9200/users/_alias/user_12' -d '{ [float] [[deleting]] -=== Delete a single index alias +=== Delete aliases -Th API to delete a single index alias, has options: + +The rest endpoint is: `/{index}/_alias/{name}` + +where [horizontal] -`index`:: The index the alias is in, the needs to be deleted. This is - a required option. -`alias`:: The name of the alias to delete. This is a required option. +`index`:: `* | _all | glob pattern | name1, name2, …` +`name`:: `* | _all | glob pattern | name1, name2, …` -The rest endpoint is: `/{index}/_alias/{alias}`. Example: +Alternatively you can use the plural `_aliases`. Example: [source,js] -------------------------------------------------- diff --git a/docs/reference/indices/delete-mapping.asciidoc b/docs/reference/indices/delete-mapping.asciidoc index 6f066dbfdce..9b72d9cb02f 100644 --- a/docs/reference/indices/delete-mapping.asciidoc +++ b/docs/reference/indices/delete-mapping.asciidoc @@ -1,8 +1,25 @@ [[indices-delete-mapping]] == Delete Mapping -Allow to delete a mapping (type) along with its data. The REST endpoint -is `/{index}/{type}` with `DELETE` method. +Allow to delete a mapping (type) along with its data. The REST endpoints are + +[source,js] +-------------------------------------------------- + +[DELETE] /{index}/{type} + +[DELETE] /{index}/{type}/_mapping + +[DELETE] /{index}/_mapping/{type} + +-------------------------------------------------- + +where + +[horizontal] + +`index`:: `* | _all | glob pattern | name1, name2, …` +`type`:: `* | _all | glob pattern | name1, name2, …` Note, most times, it make more sense to reindex the data into a fresh index compared to delete large chunks of it. diff --git a/docs/reference/indices/put-mapping.asciidoc b/docs/reference/indices/put-mapping.asciidoc index b55c19096d4..77486259001 100644 --- a/docs/reference/indices/put-mapping.asciidoc +++ b/docs/reference/indices/put-mapping.asciidoc @@ -59,3 +59,25 @@ $ curl -XPUT 'http://localhost:9200/kimchy,elasticsearch/tweet/_mapping' -d ' } ' -------------------------------------------------- + +All options: + +[source,js] +-------------------------------------------------- + +PUT /{index}/_mapping/{type} + + +-------------------------------------------------- + + +where + +[horizontal] +`{index}`:: `blank | * | _all | glob pattern | name1, name2, …` + +`{type}`:: Name of the type to add. Must be the name of the type defined in the body. + + +Instead of `_mapping` you can also use the plural `_mappings`. +The uri `PUT /{index}/{type}/_mapping` is still supported for backwardscompatibility. diff --git a/docs/reference/indices/warmers.asciidoc b/docs/reference/indices/warmers.asciidoc index abe16cb794c..bc710946512 100644 --- a/docs/reference/indices/warmers.asciidoc +++ b/docs/reference/indices/warmers.asciidoc @@ -112,25 +112,55 @@ curl -XPUT localhost:9200/test/type1/_warmer/warmer_1 -d '{ }' -------------------------------------------------- -[float] -[[removing]] -=== Delete Warmer - -Removing a warmer can be done against an index (or alias / indices) -based on its name. The provided name can be a simple wildcard expression -or omitted to remove all warmers. Some samples: +All options: [source,js] -------------------------------------------------- -# delete warmer named warmer_1 on test index -curl -XDELETE localhost:9200/test/_warmer/warmer_1 -# delete all warmers that start with warm on test index -curl -XDELETE localhost:9200/test/_warmer/warm* +PUT _warmer/{warmer_name} + +PUT /{index}/_warmer/{warmer_name} + +PUT /{index}/{type}/_warmer/{warmer_name} -# delete all warmers for test index -curl -XDELETE localhost:9200/test/_warmer/ -------------------------------------------------- + + +where + +[horizontal] +`{index}`:: `* | _all | glob pattern | name1, name2, …` + +`{type}`:: `* | _all | glob pattern | name1, name2, …` + +Instead of `_warmer` you can also use the plural `_warmers`. + + + +[float] +[[removing]] +=== Delete Warmers + +Warmers can be deleted using the following endpoint: + + + +[source,js] +-------------------------------------------------- + +[DELETE] /{index}/_warmer/{name} + +-------------------------------------------------- + + +where + +[horizontal] +`{index}`:: `* | _all | glob pattern | name1, name2, …` + +`{name}`:: `* | _all | glob pattern | name1, name2, …` + +Instead of `_warmer` you can also use the plural `_warmers`. [float] [[warmer-retrieving]] diff --git a/rest-api-spec/api/indices.delete_alias.json b/rest-api-spec/api/indices.delete_alias.json index f930e64e380..03686046820 100644 --- a/rest-api-spec/api/indices.delete_alias.json +++ b/rest-api-spec/api/indices.delete_alias.json @@ -4,7 +4,7 @@ "methods": ["DELETE"], "url": { "path": "/{index}/_alias/{name}", - "paths": ["/{index}/_alias/{name}"], + "paths": ["/{index}/_alias/{name}", "/{index}/_aliases/{name}"], "parts": { "index": { "type" : "string", diff --git a/rest-api-spec/api/indices.delete_mapping.json b/rest-api-spec/api/indices.delete_mapping.json index d6a159e8761..6494b3231d9 100644 --- a/rest-api-spec/api/indices.delete_mapping.json +++ b/rest-api-spec/api/indices.delete_mapping.json @@ -4,7 +4,7 @@ "methods": ["DELETE"], "url": { "path": "/{index}/{type}/_mapping", - "paths": ["/{index}/{type}/_mapping", "/{index}/{type}"], + "paths": ["/{index}/{type}/_mapping", "/{index}/{type}", "/{index}/_mapping/{type}", "/{index}/{type}/_mappings", "/{index}/_mappings/{type}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/api/indices.delete_warmer.json b/rest-api-spec/api/indices.delete_warmer.json index f8bbc5d29f7..be747cbeeec 100644 --- a/rest-api-spec/api/indices.delete_warmer.json +++ b/rest-api-spec/api/indices.delete_warmer.json @@ -3,8 +3,8 @@ "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html", "methods": ["DELETE"], "url": { - "path": "/{index}/_warmer", - "paths": ["/{index}/_warmer", "/{index}/_warmer/{name}", "/{index}/{type}/_warmer/{name}"], + "path": "/{index}/_warmer/{name}", + "paths": ["/{index}/_warmer", "/{index}/_warmer/{name}", "/{index}/_warmers", "/{index}/_warmers/{name}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/api/indices.put_alias.json b/rest-api-spec/api/indices.put_alias.json index cbb4c1a1981..db1072e4720 100644 --- a/rest-api-spec/api/indices.put_alias.json +++ b/rest-api-spec/api/indices.put_alias.json @@ -1,10 +1,10 @@ { "indices.put_alias": { "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html", - "methods": ["PUT"], + "methods": ["PUT", "POST"], "url": { "path": "/{index}/_alias/{name}", - "paths": ["/{index}/_alias/{name}", "/_alias/{name}", "/{index}/_alias", "/_alias"], + "paths": ["/{index}/_alias/{name}", "/_alias/{name}", "/{index}/_aliases/{name}", "/_aliases/{name}"], "parts": { "index": { "type" : "string", diff --git a/rest-api-spec/api/indices.put_mapping.json b/rest-api-spec/api/indices.put_mapping.json index 5056feaacc8..eadba64f3ce 100644 --- a/rest-api-spec/api/indices.put_mapping.json +++ b/rest-api-spec/api/indices.put_mapping.json @@ -4,11 +4,11 @@ "methods": ["PUT", "POST"], "url": { "path": "/{index}/{type}/_mapping", - "paths": ["/{index}/{type}/_mapping"], + "paths": ["/{index}/{type}/_mapping", "/{index}/_mapping/{type}", "/_mapping/{type}", "/{index}/{type}/_mappings", "/{index}/_mappings/{type}", "/_mappings/{type}"], "parts": { "index": { "type" : "list", - "required" : true, + "required" : false, "description" : "A comma-separated list of index names; use `_all` to perform the operation on all indices" }, "type": { diff --git a/rest-api-spec/api/indices.put_warmer.json b/rest-api-spec/api/indices.put_warmer.json index 7b9a8ab857e..dedb425c585 100644 --- a/rest-api-spec/api/indices.put_warmer.json +++ b/rest-api-spec/api/indices.put_warmer.json @@ -1,10 +1,10 @@ { "indices.put_warmer": { "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html", - "methods": ["PUT"], + "methods": ["PUT", "POST"], "url": { "path": "/{index}/_warmer/{name}", - "paths": ["/{index}/_warmer/{name}", "/{index}/{type}/_warmer/{name}"], + "paths": ["/_warmer/{name}", "/{index}/_warmer/{name}", "/{index}/{type}/_warmer/{name}", "/_warmers/{name}", "/{index}/_warmers/{name}", "/{index}/{type}/_warmers/{name}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/test/indices.delete_alias/all_path_options.yaml b/rest-api-spec/test/indices.delete_alias/all_path_options.yaml new file mode 100644 index 00000000000..bd158c39271 --- /dev/null +++ b/rest-api-spec/test/indices.delete_alias/all_path_options.yaml @@ -0,0 +1,225 @@ +--- +setup: + + - do: + indices.create: + index: test_index1 + + - do: + indices.create: + index: test_index2 + + - do: + indices.create: + index: foo + + - do: + indices.put_alias: + name: alias1 + body: + routing: "routing value" + - do: + indices.put_alias: + name: alias2 + body: + routing: "routing value" + +--- +"check setup": + - do: + indices.get_alias: + name: alias1 + + - match: {test_index1.aliases.alias1.search_routing: "routing value"} + - match: {test_index2.aliases.alias1.search_routing: "routing value"} + - match: {foo.aliases.alias1.search_routing: "routing value"} + + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check delete with _all index": + - do: + indices.delete_alias: + index: _all + name: alias1 + + - do: + catch: missing + indices.get_alias: + name: alias1 + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check delete with * index": + - do: + indices.delete_alias: + index: "*" + name: alias1 + + - do: + catch: missing + indices.get_alias: + name: alias1 + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check delete with index list": + - do: + indices.delete_alias: + index: "test_index1,test_index2" + name: alias1 + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check delete with prefix* index": + - do: + indices.delete_alias: + index: "test_*" + name: alias1 + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + + +--- +"check delete with index list and * aliases": + - do: + indices.delete_alias: + index: "test_index1,test_index2" + name: "*" + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {foo.aliases.alias2.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + +--- +"check delete with index list and _all aliases": + - do: + indices.delete_alias: + index: "test_index1,test_index2" + name: _all + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {foo.aliases.alias2.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + +--- +"check delete with index list and wildcard aliases": + - do: + indices.delete_alias: + index: "test_index1,test_index2" + name: "*1" + + - do: + indices.get_alias: + name: alias1 + + - match: {foo.aliases.alias1.search_routing: "routing value"} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_alias: + name: alias2 + + - match: {test_index1.aliases.alias2.search_routing: "routing value"} + - match: {test_index2.aliases.alias2.search_routing: "routing value"} + - match: {foo.aliases.alias2.search_routing: "routing value"} + +--- +"check 404 on no matching alias": + - do: + catch: missing + indices.delete_alias: + index: "*" + name: "non_existent" + + - do: + catch: missing + indices.delete_alias: + index: "non_existent" + name: "alias1" + + +--- +"check delete with blank index and blank alias": + - do: + catch: param + indices.delete_alias: + name: "alias1" + + - do: + catch: param + indices.delete_alias: + index: "test_index1" + diff --git a/rest-api-spec/test/indices.delete_mapping/all_path_options.yaml b/rest-api-spec/test/indices.delete_mapping/all_path_options.yaml new file mode 100644 index 00000000000..9fda1db4072 --- /dev/null +++ b/rest-api-spec/test/indices.delete_mapping/all_path_options.yaml @@ -0,0 +1,286 @@ +setup: + + - do: + indices.create: + index: test_index1 + body: + mappings: { test_type1: { }} + + - do: + indices.create: + index: test_index2 + body: + mappings: { test_type2: { }} + - do: + indices.create: + index: foo + body: + mappings: { test_type2: { }} + +--- +"delete with _all index": + - do: + indices.delete_mapping: + index: _all + type: test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_false: '' + +--- +"delete with * index": + - do: + indices.delete_mapping: + index: '*' + type: test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_false: '' + +--- +"delete with prefix* index": + - do: + indices.delete_mapping: + index: test* + type: test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + +--- +"delete with list of indices": + + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + +--- +"delete with index list and _all type": + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: _all + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_false: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + + +--- +"delete with index list and * type": + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: '*' + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_false: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + + +--- +"delete with index list and prefix* type": + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: '*2' + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_true: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + +--- +"delete with index list and list of types": + - do: + indices.delete_mapping: + index: test_index1,test_index2 + type: test_type1,test_type2 + + - do: + indices.exists_type: + index: test_index1 + type: test_type1 + + - is_false: '' + + - do: + indices.exists_type: + index: test_index2 + type: test_type2 + + - is_false: '' + + - do: + indices.exists_type: + index: foo + type: test_type2 + + - is_true: '' + +--- +"check 404 on no matching type": + - do: + catch: missing + indices.delete_mapping: + index: "*" + type: "non_existent" + + - do: + catch: missing + indices.delete_mapping: + index: "non_existent" + type: "test_type1" + + +--- +"check delete with blank index and blank alias": + - do: + catch: param + indices.delete_alias: + name: "alias1" + + - do: + catch: param + indices.delete_alias: + index: "test_index1" + + +--- +"check delete with blank index and blank type": + - do: + catch: param + indices.delete_mapping: + name: "test_type1" + + - do: + catch: param + indices.delete_mapping: + index: "test_index1" + diff --git a/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml b/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml new file mode 100644 index 00000000000..2953a096435 --- /dev/null +++ b/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml @@ -0,0 +1,214 @@ +setup: + - do: + indices.create: + index: test_index1 + + - do: + indices.create: + index: test_index2 + + - do: + indices.create: + index: foo + + - do: + indices.put_warmer: + index: "test_index1,test_index2,foo" + name: test_warmer1 + body: + query: + match_all: {} + + - do: + indices.put_warmer: + index: "test_index1,test_index2,foo" + name: test_warmer2 + body: + query: + match_all: {} + +--- +"Check setup": + + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.test_warmer1.source.query.match_all: {}} + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer1.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + + +--- +"check delete with _all index": + - do: + indices.delete_warmer: + index: _all + name: test_warmer1 + + - do: + catch: missing + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + +--- +"check delete with * index": + - do: + indices.delete_warmer: + index: "*" + name: test_warmer1 + + - do: + catch: missing + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + +--- +"check delete with index list": + - do: + indices.delete_warmer: + index: "test_index1,test_index2" + name: test_warmer1 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + +--- +"check delete with prefix* index": + - do: + indices.delete_warmer: + index: "test_*" + name: test_warmer1 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + + +--- +"check delete with index list and * warmers": + - do: + indices.delete_warmer: + index: "test_index1,test_index2" + name: "*" + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + +--- +"check delete with index list and _all warmers": + - do: + indices.delete_warmer: + index: "test_index1,test_index2" + name: _all + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + +--- +"check delete with index list and wildcard warmers": + - do: + indices.delete_warmer: + index: "test_index1,test_index2" + name: "*1" + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer1' } + + - match: {foo.warmers.test_warmer1.source.query.match_all: {}} + - is_false: test_index1 + - is_false: test_index2 + + - do: + indices.get_warmer: { index: _all, name: 'test_warmer2' } + + - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} + - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} + - match: {foo.warmers.test_warmer2.source.query.match_all: {}} + +--- +"check 404 on no matching test_warmer": + - do: + catch: missing + indices.delete_warmer: + index: "*" + name: "non_existent" + + - do: + catch: missing + indices.delete_warmer: + index: "non_existent" + name: "test_warmer1" + + +--- +"check delete with blank index and blank test_warmer": + - do: + catch: param + indices.delete_warmer: + name: "test_warmer1" + + - do: + catch: param + indices.delete_warmer: + index: "test_index1" + diff --git a/rest-api-spec/test/indices.put_alias/all_path_options.yaml b/rest-api-spec/test/indices.put_alias/all_path_options.yaml new file mode 100644 index 00000000000..024f6b64451 --- /dev/null +++ b/rest-api-spec/test/indices.put_alias/all_path_options.yaml @@ -0,0 +1,127 @@ +--- +setup: +# create three indices + + - do: + indices.create: + index: test_index1 + - do: + indices.create: + index: test_index2 + - do: + indices.create: + index: foo + +--- +"put alias per index": + + - do: + indices.put_alias: + index: test_index1 + name: alias + - do: + indices.put_alias: + index: test_index2 + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + + - match: {test_index2.aliases.alias: {}} + + - is_false: foo + +--- +"put alias in _all index": + + - do: + indices.put_alias: + index: _all + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - match: {foo.aliases.alias: {}} + +--- +"put alias in * index": + + + - do: + indices.put_alias: + index: '*' + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - match: {foo.aliases.alias: {}} + +--- +"put alias prefix* index": + + + - do: + indices.put_alias: + index: "test_*" + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - is_false: foo + +--- +"put alias in list of indices": + + + - do: + indices.put_alias: + index: "test_index1,test_index2" + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - is_false: foo + +--- +"put alias with blank index": + + + - do: + indices.put_alias: + name: alias + + - do: + indices.get_alias: + name: alias + + - match: {test_index1.aliases.alias: {}} + - match: {test_index2.aliases.alias: {}} + - match: {foo.aliases.alias: {}} + +--- +"put alias with mising name": + + + - do: + catch: param + indices.put_alias: {} + diff --git a/rest-api-spec/test/indices.put_mapping/all_path_options.yaml b/rest-api-spec/test/indices.put_mapping/all_path_options.yaml new file mode 100644 index 00000000000..db0df345c5d --- /dev/null +++ b/rest-api-spec/test/indices.put_mapping/all_path_options.yaml @@ -0,0 +1,178 @@ +setup: + - do: + indices.create: + index: test_index1 + - do: + indices.create: + index: test_index2 + - do: + indices.create: + index: foo + + +--- +"put one mapping per index": + - do: + indices.put_mapping: + index: test_index1 + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + - do: + indices.put_mapping: + index: test_index2 + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo: {}} + +--- +"put mapping in _all index": + + - do: + indices.put_mapping: + index: _all + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo.test_type.properties.text.type: string} + - match: {foo.test_type.properties.text.analyzer: whitespace} + +--- +"put mapping in * index": + - do: + indices.put_mapping: + index: "*" + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo.test_type.properties.text.type: string} + - match: {foo.test_type.properties.text.analyzer: whitespace} + +--- +"put mapping in prefix* index": + - do: + indices.put_mapping: + index: "test_index*" + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo: {}} + +--- +"put mapping in list of indices": + - do: + indices.put_mapping: + index: [test_index1, test_index2] + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo: {}} + +--- +"put mapping with blank index": + - do: + indices.put_mapping: + type: test_type + body: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.get_mapping: {} + + - match: {test_index1.test_type.properties.text.type: string} + - match: {test_index1.test_type.properties.text.analyzer: whitespace} + + - match: {test_index2.test_type.properties.text.type: string} + - match: {test_index2.test_type.properties.text.analyzer: whitespace} + + - match: {foo.test_type.properties.text.type: string} + - match: {foo.test_type.properties.text.analyzer: whitespace} + +--- +"put mapping with mising type": + + + - do: + catch: param + indices.put_mapping: {} + diff --git a/rest-api-spec/test/indices.put_settings/all_path_options.yaml b/rest-api-spec/test/indices.put_settings/all_path_options.yaml new file mode 100644 index 00000000000..b9ae7122216 --- /dev/null +++ b/rest-api-spec/test/indices.put_settings/all_path_options.yaml @@ -0,0 +1,113 @@ +setup: + - do: + indices.create: + index: test_index1 + - do: + indices.create: + index: test_index2 + - do: + indices.create: + index: foo + + +--- +"put settings per index": + - do: + indices.put_settings: + index: test_index1 + body: + refresh_interval: 1s + + - do: + indices.put_settings: + index: test_index2 + body: + refresh_interval: 1s + + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - is_false: foo.settings.index.refresh_interval + +--- +"put settings in _all index": + - do: + indices.put_settings: + index: _all + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - match: {foo.settings.index.refresh_interval: 1s} + +--- +"put settings in * index": + - do: + indices.put_settings: + index: '*' + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - match: {foo.settings.index.refresh_interval: 1s} + + +--- +"put settings in prefix* index": + - do: + indices.put_settings: + index: 'test*' + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - is_false: foo.settings.index.refresh_interval + +--- +"put settings in list of indices": + - skip: + version: 1 - 999 + reason: list of indices not implemented yet + - do: + indices.put_settings: + index: test_index1, test_index2 + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - is_false: foo.settings.index.refresh_interval + + +--- +"put settings in blank index": + - do: + indices.put_settings: + body: + refresh_interval: 1s + + - do: + indices.get_settings: {} + + - match: {test_index1.settings.index.refresh_interval: 1s} + - match: {test_index2.settings.index.refresh_interval: 1s} + - match: {foo.settings.index.refresh_interval: 1s} + diff --git a/rest-api-spec/test/indices.put_warmer/10_basic.yaml b/rest-api-spec/test/indices.put_warmer/10_basic.yaml index 5bcac146b68..0e2a6223605 100644 --- a/rest-api-spec/test/indices.put_warmer/10_basic.yaml +++ b/rest-api-spec/test/indices.put_warmer/10_basic.yaml @@ -32,6 +32,7 @@ - do: indices.delete_warmer: index: test_index + name: test_warmer - do: catch: missing diff --git a/rest-api-spec/test/indices.put_warmer/all_path_options.yaml b/rest-api-spec/test/indices.put_warmer/all_path_options.yaml new file mode 100644 index 00000000000..1c884e38948 --- /dev/null +++ b/rest-api-spec/test/indices.put_warmer/all_path_options.yaml @@ -0,0 +1,128 @@ +--- +setup: + + - do: + indices.create: + index: test_index1 + - do: + indices.create: + index: test_index2 + - do: + indices.create: + index: foo + +--- +"put warmer per index": + + - do: + indices.put_warmer: + index: test_index1 + name: warmer + body: + query: + match_all: {} + - do: + indices.put_warmer: + index: test_index2 + name: warmer + body: + query: + match_all: {} + + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - is_false: foo + +--- +"put warmer in _all index": + - do: + indices.put_warmer: + index: _all + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - match: {foo.warmers.warmer.source.query.match_all: {}} + +--- +"put warmer in * index": + - do: + indices.put_warmer: + index: "*" + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - match: {foo.warmers.warmer.source.query.match_all: {}} + +--- +"put warmer prefix* index": + - do: + indices.put_warmer: + index: "test_index*" + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - is_false: foo + +--- +"put warmer in list of indices": + - do: + indices.put_warmer: + index: [test_index1, test_index2] + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - is_false: foo + +--- +"put warmer with blank index": + - do: + indices.put_warmer: + name: warmer + body: + query: + match_all: {} + - do: + indices.get_warmer: { index: _all, name: '*' } + + - match: {test_index1.warmers.warmer.source.query.match_all: {}} + - match: {test_index2.warmers.warmer.source.query.match_all: {}} + - match: {foo.warmers.warmer.source.query.match_all: {}} + +--- +"put warmer with mising name": + + + - do: + catch: param + indices.put_warmer: {} + + + + diff --git a/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java index 11f8a9bde36..af6d19343bf 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java @@ -19,21 +19,25 @@ package org.elasticsearch.action.admin.indices.alias; +import com.carrotsearch.hppc.cursors.ObjectCursor; +import com.google.common.collect.ImmutableList; import com.google.common.collect.Lists; -import org.elasticsearch.ElasticsearchGenerationException; import org.elasticsearch.action.ActionRequestValidationException; +import org.elasticsearch.action.support.IndicesOptions; import org.elasticsearch.action.support.master.AcknowledgedRequest; import org.elasticsearch.cluster.metadata.AliasAction; +import org.elasticsearch.cluster.metadata.AliasAction.Type; +import org.elasticsearch.cluster.metadata.AliasMetaData; +import org.elasticsearch.cluster.metadata.MetaData; import org.elasticsearch.common.Strings; +import org.elasticsearch.common.collect.ImmutableOpenMap; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; -import org.elasticsearch.common.xcontent.ToXContent; -import org.elasticsearch.common.xcontent.XContentBuilder; -import org.elasticsearch.common.xcontent.XContentFactory; -import org.elasticsearch.common.xcontent.XContentType; +import org.elasticsearch.common.util.CollectionUtils; import org.elasticsearch.index.query.FilterBuilder; import java.io.IOException; +import java.util.ArrayList; import java.util.List; import java.util.Locale; import java.util.Map; @@ -46,115 +50,260 @@ import static org.elasticsearch.cluster.metadata.AliasAction.readAliasAction; */ public class IndicesAliasesRequest extends AcknowledgedRequest { - private List aliasActions = Lists.newArrayList(); + private List allAliasActions = Lists.newArrayList(); + + private IndicesOptions indicesOptions = IndicesOptions.fromOptions(false, false, true, false); public IndicesAliasesRequest() { } + /* + * Aliases can be added by passing multiple indices to the Request and + * deleted by passing multiple indices and aliases. They are expanded into + * distinct AliasAction instances when the request is processed. This class + * holds the AliasAction and in addition the arrays or alias names and + * indices that is later used to create the final AliasAction instances. + */ + public static class AliasActions { + private String[] indices = Strings.EMPTY_ARRAY; + private String[] aliases = Strings.EMPTY_ARRAY; + private AliasAction aliasAction; + + public AliasActions(AliasAction.Type type, String[] indices, String[] aliases) { + aliasAction = new AliasAction(type); + indices(indices); + aliases(aliases); + } + + public AliasActions(AliasAction.Type type, String index, String alias) { + aliasAction = new AliasAction(type); + indices(index); + aliases(alias); + } + + AliasActions(AliasAction.Type type, String[] index, String alias) { + aliasAction = new AliasAction(type); + indices(index); + aliases(alias); + } + + public AliasActions(AliasAction action) { + this.aliasAction = action; + indices(action.index()); + aliases(action.alias()); + } + + public AliasActions(Type type, String index, String[] aliases) { + aliasAction = new AliasAction(type); + indices(index); + aliases(aliases); + } + + public AliasActions() { + } + + public AliasActions filter(Map filter) { + aliasAction.filter(filter); + return this; + } + + public AliasActions filter(FilterBuilder filter) { + aliasAction.filter(filter); + return this; + } + + public Type actionType() { + return aliasAction.actionType(); + } + + public void routing(String routing) { + aliasAction.routing(routing); + } + + public void searchRouting(String searchRouting) { + aliasAction.searchRouting(searchRouting); + } + + public void indexRouting(String indexRouting) { + aliasAction.indexRouting(indexRouting); + } + + public AliasActions filter(String filter) { + aliasAction.filter(filter); + return this; + } + + public void indices(String... indices) { + List finalIndices = new ArrayList(); + for (String index : indices) { + if (index != null) { + finalIndices.add(index); + } + } + this.indices = finalIndices.toArray(new String[finalIndices.size()]); + } + + public void aliases(String... aliases) { + this.aliases = aliases; + } + + public String[] aliases() { + return aliases; + } + + public String[] indices() { + return indices; + } + + public AliasAction aliasAction() { + return aliasAction; + } + + public String[] concreteAliases(MetaData metaData, String concreteIndex) { + if (aliasAction.actionType() == Type.REMOVE) { + //for DELETE we expand the aliases + String[] indexAsArray = {concreteIndex}; + ImmutableOpenMap> aliasMetaData = metaData.findAliases(aliases, indexAsArray); + List finalAliases = new ArrayList (); + for (ObjectCursor> curAliases : aliasMetaData.values()) { + for (AliasMetaData aliasMeta: curAliases.value) { + finalAliases.add(aliasMeta.alias()); + } + } + return finalAliases.toArray(new String[finalAliases.size()]); + } else { + //for add we just return the current aliases + return aliases; + } + } + public AliasActions readFrom(StreamInput in) throws IOException { + indices = in.readStringArray(); + aliases = in.readStringArray(); + aliasAction = readAliasAction(in); + return this; + } + + public void writeTo(StreamOutput out) throws IOException { + out.writeStringArray(indices); + out.writeStringArray(aliases); + this.aliasAction.writeTo(out); + } + } + /** * Adds an alias to the index. - * - * @param index The index * @param alias The alias + * @param indices The indices */ - public IndicesAliasesRequest addAlias(String index, String alias) { - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias)); + public IndicesAliasesRequest addAlias(String alias, String... indices) { + addAliasAction(new AliasActions(AliasAction.Type.ADD, indices, alias)); + return this; + } + + + public void addAliasAction(AliasActions aliasAction) { + allAliasActions.add(aliasAction); + } + + + public IndicesAliasesRequest addAliasAction(AliasAction action) { + addAliasAction(new AliasActions(action)); + return this; + } + + /** + * Adds an alias to the index. + * @param alias The alias + * @param filter The filter + * @param indices The indices + */ + public IndicesAliasesRequest addAlias(String alias, Map filter, String... indices) { + addAliasAction(new AliasActions(AliasAction.Type.ADD, indices, alias).filter(filter)); return this; } /** * Adds an alias to the index. - * - * @param index The index - * @param alias The alias - * @param filter The filter - */ - public IndicesAliasesRequest addAlias(String index, String alias, String filter) { - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias, filter)); - return this; - } - - /** - * Adds an alias to the index. - * - * @param index The index - * @param alias The alias - * @param filter The filter - */ - public IndicesAliasesRequest addAlias(String index, String alias, Map filter) { - if (filter == null || filter.isEmpty()) { - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias)); - return this; - } - try { - XContentBuilder builder = XContentFactory.contentBuilder(XContentType.JSON); - builder.map(filter); - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias, builder.string())); - return this; - } catch (IOException e) { - throw new ElasticsearchGenerationException("Failed to generate [" + filter + "]", e); - } - } - - /** - * Adds an alias to the index. - * - * @param index The index * @param alias The alias * @param filterBuilder The filter + * @param indices The indices */ - public IndicesAliasesRequest addAlias(String index, String alias, FilterBuilder filterBuilder) { - if (filterBuilder == null) { - aliasActions.add(new AliasAction(AliasAction.Type.ADD, index, alias)); - return this; - } - try { - XContentBuilder builder = XContentFactory.jsonBuilder(); - filterBuilder.toXContent(builder, ToXContent.EMPTY_PARAMS); - builder.close(); - return addAlias(index, alias, builder.string()); - } catch (IOException e) { - throw new ElasticsearchGenerationException("Failed to build json for alias request", e); - } + public IndicesAliasesRequest addAlias(String alias, FilterBuilder filterBuilder, String... indices) { + addAliasAction(new AliasActions(AliasAction.Type.ADD, indices, alias).filter(filterBuilder)); + return this; } - + + + /** + * Removes an alias to the index. + * + * @param indices The indices + * @param aliases The aliases + */ + public IndicesAliasesRequest removeAlias(String[] indices, String... aliases) { + addAliasAction(new AliasActions(AliasAction.Type.REMOVE, indices, aliases)); + return this; + } + /** * Removes an alias to the index. * * @param index The index - * @param alias The alias + * @param aliases The aliases */ - public IndicesAliasesRequest removeAlias(String index, String alias) { - aliasActions.add(new AliasAction(AliasAction.Type.REMOVE, index, alias)); + public IndicesAliasesRequest removeAlias(String index, String... aliases) { + addAliasAction(new AliasActions(AliasAction.Type.REMOVE, index, aliases)); return this; } - public IndicesAliasesRequest addAliasAction(AliasAction action) { - aliasActions.add(action); - return this; + List aliasActions() { + return this.allAliasActions; } - List aliasActions() { - return this.aliasActions; - } - - public List getAliasActions() { + public List getAliasActions() { return aliasActions(); } @Override public ActionRequestValidationException validate() { ActionRequestValidationException validationException = null; - if (aliasActions.isEmpty()) { + if (allAliasActions.isEmpty()) { return addValidationError("Must specify at least one alias action", validationException); } - for (AliasAction aliasAction : aliasActions) { - if (!Strings.hasText(aliasAction.alias())) { - validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + "] requires an [alias] to be set", validationException); + for (AliasActions aliasAction : allAliasActions) { + if (aliasAction.actionType() == AliasAction.Type.ADD) { + if (aliasAction.aliases.length != 1) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "] requires exactly one [alias] to be set", validationException); + } + if (!Strings.hasText(aliasAction.aliases[0])) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "] requires an [alias] to be set", validationException); + } + } else { + if (aliasAction.aliases.length == 0) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "]: aliases may not be empty", validationException); + } + for (String alias : aliasAction.aliases) { + if (!Strings.hasText(alias)) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "]: [alias] may not be empty string", validationException); + } + } + if (CollectionUtils.isEmpty(aliasAction.indices)) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "]: indices may not be empty", validationException); + } } - if (!Strings.hasText(aliasAction.index())) { - validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + "] requires an [index] to be set", validationException); + if (!CollectionUtils.isEmpty(aliasAction.indices)) { + for (String index : aliasAction.indices) { + if (!Strings.hasText(index)) { + validationException = addValidationError("Alias action [" + aliasAction.actionType().name().toLowerCase(Locale.ENGLISH) + + "]: [index] may not be empty string", validationException); + } + } } } return validationException; @@ -165,7 +314,7 @@ public class IndicesAliasesRequest extends AcknowledgedRequest filter) { + request.addAlias(alias, filter, indices); + return this; + } + + /** + * Adds an alias to the index. + * + * @param index The indices * @param alias The alias * @param filter The filter */ public IndicesAliasesRequestBuilder addAlias(String index, String alias, Map filter) { - request.addAlias(index, alias, filter); + request.addAlias(alias, filter, index); return this; } /** * Adds an alias to the index. * - * @param index The index + * @param indices The indices + * @param alias The alias + * @param filterBuilder The filter + */ + public IndicesAliasesRequestBuilder addAlias(String indices[], String alias, FilterBuilder filterBuilder) { + request.addAlias(alias, filterBuilder, indices); + return this; + } + + /** + * Adds an alias to the index. + * + * @param index The index * @param alias The alias * @param filterBuilder The filter */ public IndicesAliasesRequestBuilder addAlias(String index, String alias, FilterBuilder filterBuilder) { - request.addAlias(index, alias, filterBuilder); + request.addAlias(alias, filterBuilder, index); return this; } /** - * Adds an alias action to the request. - * - * @param aliasAction The alias Action - */ - public IndicesAliasesRequestBuilder addAliasAction(AliasAction aliasAction) { - request.addAliasAction(aliasAction); - return this; - } - - /** - * Removes an alias to the index. + * Removes an alias from the index. * * @param index The index * @param alias The alias @@ -104,9 +144,54 @@ public class IndicesAliasesRequestBuilder extends AcknowledgedRequestBuilder listener) { ((IndicesAdminClient) client).aliases(request, listener); } + + /** + * Adds an alias action to the request. + * + * @param aliasAction The alias action + */ + public IndicesAliasesRequestBuilder addAliasAction(AliasAction aliasAction) { + request.addAliasAction(aliasAction); + return this; + } + + /** + * Adds an alias action to the request. + * + * @param aliasAction The alias action + */ + public IndicesAliasesRequestBuilder addAliasAction( + AliasActions action) { + request.addAliasAction(action); + return this; + } + + } diff --git a/src/main/java/org/elasticsearch/action/admin/indices/alias/TransportIndicesAliasesAction.java b/src/main/java/org/elasticsearch/action/admin/indices/alias/TransportIndicesAliasesAction.java index 5fab726c33a..64da64e5cfd 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/alias/TransportIndicesAliasesAction.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/alias/TransportIndicesAliasesAction.java @@ -22,6 +22,7 @@ package org.elasticsearch.action.admin.indices.alias; import com.google.common.collect.Sets; import org.elasticsearch.ElasticsearchException; import org.elasticsearch.action.ActionListener; +import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest.AliasActions; import org.elasticsearch.action.support.master.TransportMasterNodeOperationAction; import org.elasticsearch.cluster.ClusterService; import org.elasticsearch.cluster.ClusterState; @@ -33,9 +34,13 @@ import org.elasticsearch.cluster.metadata.AliasAction; import org.elasticsearch.cluster.metadata.MetaDataIndexAliasesService; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.rest.action.admin.indices.alias.delete.AliasesMissingException; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.transport.TransportService; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; import java.util.Set; /** @@ -76,8 +81,10 @@ public class TransportIndicesAliasesAction extends TransportMasterNodeOperationA @Override protected ClusterBlockException checkBlock(IndicesAliasesRequest request, ClusterState state) { Set indices = Sets.newHashSet(); - for (AliasAction aliasAction : request.aliasActions()) { - indices.add(aliasAction.index()); + for (AliasActions aliasAction : request.aliasActions()) { + for (String index : aliasAction.indices()) { + indices.add(index); + } } return state.blocks().indicesBlockedException(ClusterBlockLevel.METADATA, indices.toArray(new String[indices.size()])); } @@ -85,9 +92,36 @@ public class TransportIndicesAliasesAction extends TransportMasterNodeOperationA @Override protected void masterOperation(final IndicesAliasesRequest request, final ClusterState state, final ActionListener listener) throws ElasticsearchException { + //Expand the indices names + List actions = request.aliasActions(); + List finalActions = new ArrayList(); + boolean hasOnlyDeletesButNoneCanBeDone = true; + Set aliases = new HashSet(); + for (AliasActions action : actions) { + //expand indices + String[] concreteIndices = state.metaData().concreteIndices(action.indices(), request.indicesOptions()); + //collect the aliases + for (String alias : action.aliases()) { + aliases.add(alias); + } + for (String index : concreteIndices) { + for (String alias : action.concreteAliases(state.metaData(), index)) { + AliasAction finalAction = new AliasAction(action.aliasAction()); + finalAction.index(index); + finalAction.alias(alias); + finalActions.add(finalAction); + //if there is only delete requests, none will be added if the types do not map to any existing type + hasOnlyDeletesButNoneCanBeDone = false; + } + } + } + if (hasOnlyDeletesButNoneCanBeDone && actions.size() != 0) { + throw new AliasesMissingException(aliases.toArray(new String[aliases.size()])); + } + request.aliasActions().clear(); IndicesAliasesClusterStateUpdateRequest updateRequest = new IndicesAliasesClusterStateUpdateRequest() .ackTimeout(request.timeout()).masterNodeTimeout(request.masterNodeTimeout()) - .actions(request.aliasActions().toArray(new AliasAction[request.aliasActions().size()])); + .actions(finalActions.toArray(new AliasAction[finalActions.size()])); indexAliasesService.indicesAliases(updateRequest, new ClusterStateUpdateListener() { @Override diff --git a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingClusterStateUpdateRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingClusterStateUpdateRequest.java index 49ed091a19b..5a2ef82eba1 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingClusterStateUpdateRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingClusterStateUpdateRequest.java @@ -26,7 +26,7 @@ import org.elasticsearch.cluster.ack.IndicesClusterStateUpdateRequest; */ public class DeleteMappingClusterStateUpdateRequest extends IndicesClusterStateUpdateRequest { - private String type; + private String[] types; DeleteMappingClusterStateUpdateRequest() { @@ -35,15 +35,15 @@ public class DeleteMappingClusterStateUpdateRequest extends IndicesClusterStateU /** * Returns the type to be removed */ - public String type() { - return type; + public String[] types() { + return types; } /** * Sets the type to be removed */ - public DeleteMappingClusterStateUpdateRequest type(String type) { - this.type = type; + public DeleteMappingClusterStateUpdateRequest types(String[] types) { + this.types = types; return this; } } diff --git a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingRequest.java index 3d1f0fc4509..e342a5d7673 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/DeleteMappingRequest.java @@ -23,8 +23,10 @@ import org.elasticsearch.Version; import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.action.support.IndicesOptions; import org.elasticsearch.action.support.master.AcknowledgedRequest; +import org.elasticsearch.common.Strings; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.util.CollectionUtils; import java.io.IOException; @@ -37,7 +39,7 @@ public class DeleteMappingRequest extends AcknowledgedRequest() { @Override public void onResponse(FlushResponse flushResponse) { + + // get all types that need to be deleted. + ImmutableOpenMap> result = state.metaData().findMappings( + request.indices(), request.types() + ); + // create OrFilter with type filters within to account for different types + OrFilterBuilder filterBuilder = new OrFilterBuilder(); + Set types = new HashSet(); + for (ObjectObjectCursor> typesMeta : result) { + for (ObjectObjectCursor type : typesMeta.value) { + filterBuilder.add(new TypeFilterBuilder(type.key)); + types.add(type.key); + } + } + request.types(types.toArray(new String[types.size()])); QuerySourceBuilder querySourceBuilder = new QuerySourceBuilder() - .setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.typeFilter(request.type()))); + .setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), filterBuilder)); deleteByQueryAction.execute(Requests.deleteByQueryRequest(request.indices()).source(querySourceBuilder), new ActionListener() { @Override public void onResponse(DeleteByQueryResponse deleteByQueryResponse) { @@ -126,7 +148,7 @@ public class TransportDeleteMappingAction extends TransportMasterNodeOperationAc protected void removeMapping() { DeleteMappingClusterStateUpdateRequest clusterStateUpdateRequest = new DeleteMappingClusterStateUpdateRequest() - .indices(request.indices()).type(request.type()) + .indices(request.indices()).types(request.types()) .ackTimeout(request.timeout()) .masterNodeTimeout(request.masterNodeTimeout()); diff --git a/src/main/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequest.java index 90057b8fa6d..3c79b104517 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequest.java @@ -26,15 +26,18 @@ import org.elasticsearch.common.Nullable; import org.elasticsearch.common.Strings; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.util.CollectionUtils; import java.io.IOException; +import static org.elasticsearch.action.ValidateActions.addValidationError; + /** * A request to delete an index warmer. */ public class DeleteWarmerRequest extends AcknowledgedRequest { - private String name; + private String[] names = Strings.EMPTY_ARRAY; private IndicesOptions indicesOptions = IndicesOptions.fromOptions(false, false, true, false); private String[] indices = Strings.EMPTY_ARRAY; @@ -46,36 +49,60 @@ public class DeleteWarmerRequest extends AcknowledgedRequest listener) throws ElasticsearchException { - clusterService.submitStateUpdateTask("delete_warmer [" + request.name() + "]", new AckedClusterStateUpdateTask() { + clusterService.submitStateUpdateTask("delete_warmer [" + Arrays.toString(request.names()) + "]", new AckedClusterStateUpdateTask() { @Override public boolean mustAck(DiscoveryNode discoveryNode) { @@ -118,7 +119,7 @@ public class TransportDeleteWarmerAction extends TransportMasterNodeOperationAct @Override public void onFailure(String source, Throwable t) { - logger.debug("failed to delete warmer [{}] on indices [{}]", t, request.name(), request.indices()); + logger.debug("failed to delete warmer [{}] on indices [{}]", t, Arrays.toString(request.names()), request.indices()); listener.onFailure(t); } @@ -136,10 +137,16 @@ public class TransportDeleteWarmerAction extends TransportMasterNodeOperationAct if (warmers != null) { List entries = Lists.newArrayList(); for (IndexWarmersMetaData.Entry entry : warmers.entries()) { - if (request.name() == null || Regex.simpleMatch(request.name(), entry.name())) { - globalFoundAtLeastOne = true; - // don't add it... - } else { + boolean keepWarmer = true; + for (String warmer : request.names()) { + if (Regex.simpleMatch(warmer, entry.name()) || warmer.equals("_all")) { + globalFoundAtLeastOne = true; + keepWarmer = false; + // don't add it... + break; + } + } + if (keepWarmer) { entries.add(entry); } } @@ -153,11 +160,7 @@ public class TransportDeleteWarmerAction extends TransportMasterNodeOperationAct } if (!globalFoundAtLeastOne) { - if (request.name() == null) { - // full match, just return with no failure - return currentState; - } - throw new IndexWarmerMissingException(request.name()); + throw new IndexWarmerMissingException(request.names()); } if (logger.isInfoEnabled()) { @@ -169,8 +172,10 @@ public class TransportDeleteWarmerAction extends TransportMasterNodeOperationAct IndexWarmersMetaData warmers = indexMetaData.custom(IndexWarmersMetaData.TYPE); if (warmers != null) { for (IndexWarmersMetaData.Entry entry : warmers.entries()) { - if (Regex.simpleMatch(request.name(), entry.name())) { - logger.info("[{}] delete warmer [{}]", index, entry.name()); + for (String warmer : request.names()) { + if (Regex.simpleMatch(warmer, entry.name()) || warmer.equals("_all")) { + logger.info("[{}] delete warmer [{}]", index, entry.name()); + } } } } diff --git a/src/main/java/org/elasticsearch/cluster/metadata/AliasAction.java b/src/main/java/org/elasticsearch/cluster/metadata/AliasAction.java index d0834760f4d..29bcba360bb 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/AliasAction.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/AliasAction.java @@ -82,7 +82,20 @@ public class AliasAction implements Streamable { private AliasAction() { } - + + public AliasAction(AliasAction other) { + this.actionType = other.actionType; + this.index = other.index; + this.alias = other.alias; + this.filter = other.filter; + this.indexRouting = other.indexRouting; + this.searchRouting = other.searchRouting; + } + + public AliasAction(Type actionType) { + this.actionType = actionType; + } + public AliasAction(Type actionType, String index, String alias) { this.actionType = actionType; this.index = index; @@ -99,10 +112,20 @@ public class AliasAction implements Streamable { public Type actionType() { return actionType; } + + public AliasAction index(String index) { + this.index = index; + return this; + } public String index() { return index; } + + public AliasAction alias(String alias) { + this.alias = alias; + return this; + } public String alias() { return alias; @@ -181,42 +204,21 @@ public class AliasAction implements Streamable { @Override public void readFrom(StreamInput in) throws IOException { actionType = Type.fromValue(in.readByte()); - index = in.readString(); - alias = in.readString(); - if (in.readBoolean()) { - filter = in.readString(); - } - if (in.readBoolean()) { - indexRouting = in.readString(); - } - if (in.readBoolean()) { - searchRouting = in.readString(); - } + index = in.readOptionalString(); + alias = in.readOptionalString(); + filter = in.readOptionalString(); + indexRouting = in.readOptionalString(); + searchRouting = in.readOptionalString(); } @Override public void writeTo(StreamOutput out) throws IOException { out.writeByte(actionType.value()); - out.writeString(index); - out.writeString(alias); - if (filter == null) { - out.writeBoolean(false); - } else { - out.writeBoolean(true); - out.writeString(filter); - } - if (indexRouting == null) { - out.writeBoolean(false); - } else { - out.writeBoolean(true); - out.writeString(indexRouting); - } - if (searchRouting == null) { - out.writeBoolean(false); - } else { - out.writeBoolean(true); - out.writeString(searchRouting); - } + out.writeOptionalString(index); + out.writeOptionalString(alias); + out.writeOptionalString(filter); + out.writeOptionalString(indexRouting); + out.writeOptionalString(searchRouting); } public static AliasAction newAddAliasAction(String index, String alias) { diff --git a/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java b/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java index 2ac352f3643..b2310d2532c 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java @@ -279,7 +279,7 @@ public class MetaData implements Iterable { return ImmutableOpenMap.of(); } - boolean matchAllAliases = aliases.length == 0; + boolean matchAllAliases = matchAllAliases(aliases); ImmutableOpenMap.Builder> mapBuilder = ImmutableOpenMap.builder(); Iterable intersection = HppcMaps.intersection(ObjectOpenHashSet.from(concreteIndices), indices.keys()); for (String index : intersection) { @@ -298,6 +298,15 @@ public class MetaData implements Iterable { } return mapBuilder.build(); } + + private boolean matchAllAliases(final String[] aliases) { + for (String alias : aliases) { + if (alias.equals("_all")) { + return true; + } + } + return aliases.length == 0; + } /** * Checks if at least one of the specified aliases exists in the specified concrete indices. Wildcards are supported in the @@ -331,6 +340,12 @@ public class MetaData implements Iterable { return false; } + /* + * Finds all mappings for types and concrete indices. Types are expanded to + * include all types that match the glob patterns in the types array. Empty + * types array, null or {"_all"} will be expanded to all types available for + * the given indices. + */ public ImmutableOpenMap> findMappings(String[] concreteIndices, final String[] types) { assert types != null; assert concreteIndices != null; @@ -343,7 +358,7 @@ public class MetaData implements Iterable { for (String index : intersection) { IndexMetaData indexMetaData = indices.get(index); ImmutableOpenMap.Builder filteredMappings; - if (types.length == 0) { + if (isAllTypes(types)) { indexMapBuilder.put(index, indexMetaData.getMappings()); // No types specified means get it all } else { @@ -945,7 +960,18 @@ public class MetaData implements Iterable { * @return true if the provided array maps to all indices, false otherwise */ public boolean isAllIndices(String[] aliasesOrIndices) { - return aliasesOrIndices == null || aliasesOrIndices.length == 0 || isExplicitAllIndices(aliasesOrIndices); + return aliasesOrIndices == null || aliasesOrIndices.length == 0 || isExplicitAllPattern(aliasesOrIndices); + } + + /** + * Identifies whether the array containing type names given as argument refers to all types + * The empty or null array identifies all types + * + * @param types the array containing index names + * @return true if the provided array maps to all indices, false otherwise + */ + public boolean isAllTypes(String[] types) { + return types == null || types.length == 0 || isExplicitAllPattern(types); } /** @@ -955,7 +981,7 @@ public class MetaData implements Iterable { * @param aliasesOrIndices the array containing index names * @return true if the provided array explicitly maps to all indices, false otherwise */ - public boolean isExplicitAllIndices(String[] aliasesOrIndices) { + public boolean isExplicitAllPattern(String[] aliasesOrIndices) { return aliasesOrIndices != null && aliasesOrIndices.length == 1 && "_all".equals(aliasesOrIndices[0]); } diff --git a/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java b/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java index 046dd44190f..414bc40a550 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/MetaDataMappingService.java @@ -385,7 +385,7 @@ public class MetaDataMappingService extends AbstractComponent { } public void removeMapping(final DeleteMappingClusterStateUpdateRequest request, final ClusterStateUpdateListener listener) { - clusterService.submitStateUpdateTask("remove-mapping [" + request.type() + "]", Priority.HIGH, new AckedClusterStateUpdateTask() { + clusterService.submitStateUpdateTask("remove-mapping [" + Arrays.toString(request.types()) + "]", Priority.HIGH, new AckedClusterStateUpdateTask() { @Override public boolean mustAck(DiscoveryNode discoveryNode) { @@ -428,21 +428,30 @@ public class MetaDataMappingService extends AbstractComponent { String latestIndexWithout = null; for (String indexName : request.indices()) { IndexMetaData indexMetaData = currentState.metaData().index(indexName); + IndexMetaData.Builder indexBuilder = IndexMetaData.builder(indexMetaData); + if (indexMetaData != null) { - if (indexMetaData.mappings().containsKey(request.type())) { - builder.put(IndexMetaData.builder(indexMetaData).removeMapping(request.type())); - changed = true; - } else { + boolean isLatestIndexWithout = true; + for (String type : request.types()) { + if (indexMetaData.mappings().containsKey(type)) { + indexBuilder.removeMapping(type); + changed = true; + isLatestIndexWithout = false; + } + } + if (isLatestIndexWithout) { latestIndexWithout = indexMetaData.index(); } + } + builder.put(indexBuilder); } if (!changed) { - throw new TypeMissingException(new Index(latestIndexWithout), request.type()); + throw new TypeMissingException(new Index(latestIndexWithout), request.types()); } - logger.info("[{}] remove_mapping [{}]", request.indices(), request.type()); + logger.info("[{}] remove_mapping [{}]", request.indices(), request.types()); return ClusterState.builder(currentState).metaData(builder).build(); } diff --git a/src/main/java/org/elasticsearch/common/util/CollectionUtils.java b/src/main/java/org/elasticsearch/common/util/CollectionUtils.java index c24a6e67be0..9d6e2753a87 100644 --- a/src/main/java/org/elasticsearch/common/util/CollectionUtils.java +++ b/src/main/java/org/elasticsearch/common/util/CollectionUtils.java @@ -187,5 +187,16 @@ public enum CollectionUtils { } return uniqueCount; } + + /** + * Checks if the given array contains any elements. + * + * @param array The array to check + * + * @return false if the array contains an element, true if not or the array is null. + */ + public static boolean isEmpty(Object[] array) { + return array == null || array.length == 0; + } } diff --git a/src/main/java/org/elasticsearch/indices/TypeMissingException.java b/src/main/java/org/elasticsearch/indices/TypeMissingException.java index 494c2ada693..ae2830b9575 100644 --- a/src/main/java/org/elasticsearch/indices/TypeMissingException.java +++ b/src/main/java/org/elasticsearch/indices/TypeMissingException.java @@ -23,17 +23,19 @@ import org.elasticsearch.index.Index; import org.elasticsearch.index.IndexException; import org.elasticsearch.rest.RestStatus; +import java.util.Arrays; + /** * */ public class TypeMissingException extends IndexException { - public TypeMissingException(Index index, String type) { - super(index, "type[" + type + "] missing"); + public TypeMissingException(Index index, String... types) { + super(index, "type[" + Arrays.toString(types) + "] missing"); } - public TypeMissingException(Index index, String type, String message) { - super(index, "type[" + type + "] missing: " + message); + public TypeMissingException(Index index, String[] types, String message) { + super(index, "type[" + Arrays.toString(types) + "] missing: " + message); } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/AliasesMissingException.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/AliasesMissingException.java new file mode 100644 index 00000000000..dfc98b0cb82 --- /dev/null +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/AliasesMissingException.java @@ -0,0 +1,46 @@ +/* + * Licensed to Elasticsearch under one or more contributor + * license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright + * ownership. Elasticsearch licenses this file to you under + * the Apache License, Version 2.0 (the "License"); you may + * not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.elasticsearch.rest.action.admin.indices.alias.delete; + +import org.elasticsearch.ElasticsearchException; +import org.elasticsearch.rest.RestStatus; + +import java.util.Arrays; + +/** + * + */ +public class AliasesMissingException extends ElasticsearchException { + + private final String[] names; + + public AliasesMissingException(String... names) { + super("aliases [" + Arrays.toString(names) + "] missing"); + this.names = names; + } + + public String[] names() { + return this.names; + } + + @Override + public RestStatus status() { + return RestStatus.NOT_FOUND; + } +} diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java index 4e03f603c68..8cbd9d77092 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/delete/RestIndexDeleteAliasesAction.java @@ -21,6 +21,7 @@ package org.elasticsearch.rest.action.admin.indices.alias.delete; import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest; import org.elasticsearch.action.admin.indices.alias.IndicesAliasesResponse; import org.elasticsearch.client.Client; +import org.elasticsearch.common.Strings; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.rest.*; @@ -35,15 +36,16 @@ public class RestIndexDeleteAliasesAction extends BaseRestHandler { public RestIndexDeleteAliasesAction(Settings settings, Client client, RestController controller) { super(settings, client); controller.registerHandler(DELETE, "/{index}/_alias/{name}", this); + controller.registerHandler(DELETE, "/{index}/_aliases/{name}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { - final String index = request.param("index"); - String alias = request.param("name"); + final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); + final String[] aliases = Strings.splitStringByCommaToArray(request.param("name")); IndicesAliasesRequest indicesAliasesRequest = new IndicesAliasesRequest(); indicesAliasesRequest.timeout(request.paramAsTime("timeout", indicesAliasesRequest.timeout())); - indicesAliasesRequest.removeAlias(index, alias); + indicesAliasesRequest.removeAlias(indices, aliases); indicesAliasesRequest.masterNodeTimeout(request.paramAsTime("master_timeout", indicesAliasesRequest.masterNodeTimeout())); client.admin().indices().aliases(indicesAliasesRequest, new AcknowledgedRestResponseActionListener(request, channel, logger)); diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java index 3b4fa4ee828..c2e3b5d5492 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/put/RestIndexPutAliasAction.java @@ -20,9 +20,11 @@ package org.elasticsearch.rest.action.admin.indices.alias.put; import org.elasticsearch.ElasticsearchIllegalArgumentException; import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest; +import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest.AliasActions; import org.elasticsearch.action.admin.indices.alias.IndicesAliasesResponse; import org.elasticsearch.client.Client; import org.elasticsearch.cluster.metadata.AliasAction; +import org.elasticsearch.common.Strings; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.xcontent.XContentFactory; @@ -32,6 +34,7 @@ import org.elasticsearch.rest.*; import java.io.IOException; import java.util.Map; +import static org.elasticsearch.rest.RestRequest.Method.POST; import static org.elasticsearch.rest.RestRequest.Method.PUT; /** @@ -43,13 +46,22 @@ public class RestIndexPutAliasAction extends BaseRestHandler { super(settings, client); controller.registerHandler(PUT, "/{index}/_alias/{name}", this); controller.registerHandler(PUT, "/_alias/{name}", this); + controller.registerHandler(PUT, "/{index}/_aliases/{name}", this); + controller.registerHandler(PUT, "/_aliases/{name}", this); controller.registerHandler(PUT, "/{index}/_alias", this); controller.registerHandler(PUT, "/_alias", this); + + controller.registerHandler(POST, "/{index}/_alias/{name}", this); + controller.registerHandler(POST, "/_alias/{name}", this); + controller.registerHandler(POST, "/{index}/_aliases/{name}", this); + controller.registerHandler(POST, "/_aliases/{name}", this); + controller.registerHandler(PUT, "/{index}/_aliases", this); + //we cannot add POST for "/_aliases" because this is the _aliases api already defined in RestIndicesAliasesAction } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { - String index = request.param("index"); + String[] indices = Strings.splitStringByCommaToArray(request.param("index")); String alias = request.param("name"); Map filter = null; String routing = null; @@ -70,7 +82,7 @@ public class RestIndexPutAliasAction extends BaseRestHandler { currentFieldName = parser.currentName(); } else if (token.isValue()) { if ("index".equals(currentFieldName)) { - index = parser.text(); + indices = Strings.splitStringByCommaToArray(parser.text()); } else if ("alias".equals(currentFieldName)) { alias = parser.text(); } else if ("routing".equals(currentFieldName)) { @@ -102,9 +114,11 @@ public class RestIndexPutAliasAction extends BaseRestHandler { IndicesAliasesRequest indicesAliasesRequest = new IndicesAliasesRequest(); indicesAliasesRequest.timeout(request.paramAsTime("timeout", indicesAliasesRequest.timeout())); - AliasAction aliasAction = new AliasAction(AliasAction.Type.ADD, index, alias); + String[] aliases = new String[] {alias}; + IndicesAliasesRequest.AliasActions aliasAction = new AliasActions(AliasAction.Type.ADD, indices, aliases); indicesAliasesRequest.addAliasAction(aliasAction); indicesAliasesRequest.masterNodeTimeout(request.paramAsTime("master_timeout", indicesAliasesRequest.masterNodeTimeout())); + if (routing != null) { aliasAction.routing(routing); diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java index 28cff7e7763..d915609b243 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/delete/RestDeleteMappingAction.java @@ -41,13 +41,18 @@ public class RestDeleteMappingAction extends BaseRestHandler { super(settings, client); controller.registerHandler(DELETE, "/{index}/{type}/_mapping", this); controller.registerHandler(DELETE, "/{index}/{type}", this); + controller.registerHandler(DELETE, "/{index}/_mapping/{type}", this); + + //support _mappings also + controller.registerHandler(DELETE, "/{index}/{type}/_mappings", this); + controller.registerHandler(DELETE, "/{index}/_mappings/{type}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { DeleteMappingRequest deleteMappingRequest = deleteMappingRequest(Strings.splitStringByCommaToArray(request.param("index"))); deleteMappingRequest.listenerThreaded(false); - deleteMappingRequest.type(request.param("type")); + deleteMappingRequest.types(Strings.splitStringByCommaToArray(request.param("type"))); deleteMappingRequest.timeout(request.paramAsTime("timeout", deleteMappingRequest.timeout())); deleteMappingRequest.masterNodeTimeout(request.paramAsTime("master_timeout", deleteMappingRequest.masterNodeTimeout())); deleteMappingRequest.indicesOptions(IndicesOptions.fromRequest(request, deleteMappingRequest.indicesOptions())); diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java index e1f285d527d..b6e9a5ebabc 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/put/RestPutMappingAction.java @@ -37,14 +37,30 @@ import static org.elasticsearch.rest.RestRequest.Method.PUT; */ public class RestPutMappingAction extends BaseRestHandler { + @Inject public RestPutMappingAction(Settings settings, Client client, RestController controller) { super(settings, client); - controller.registerHandler(PUT, "/{index}/_mapping", this); + controller.registerHandler(PUT, "/{index}/_mapping/", this); controller.registerHandler(PUT, "/{index}/{type}/_mapping", this); + controller.registerHandler(PUT, "/{index}/_mapping/{type}", this); + controller.registerHandler(PUT, "/_mapping/{type}", this); - controller.registerHandler(POST, "/{index}/_mapping", this); + controller.registerHandler(POST, "/{index}/_mapping/", this); controller.registerHandler(POST, "/{index}/{type}/_mapping", this); + controller.registerHandler(POST, "/{index}/_mapping/{type}", this); + controller.registerHandler(POST, "/_mapping/{type}", this); + + //register the same paths, but with plural form _mappings + controller.registerHandler(PUT, "/{index}/_mappings/", this); + controller.registerHandler(PUT, "/{index}/{type}/_mappings", this); + controller.registerHandler(PUT, "/{index}/_mappings/{type}", this); + controller.registerHandler(PUT, "/_mappings/{type}", this); + + controller.registerHandler(POST, "/{index}/_mappings/", this); + controller.registerHandler(POST, "/{index}/{type}/_mappings", this); + controller.registerHandler(POST, "/{index}/_mappings/{type}", this); + controller.registerHandler(POST, "/_mappings/{type}", this); } @Override diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java index 2ff34c8f28a..e54e4589921 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/delete/RestDeleteWarmerAction.java @@ -37,12 +37,13 @@ public class RestDeleteWarmerAction extends BaseRestHandler { super(settings, client); controller.registerHandler(DELETE, "/{index}/_warmer", this); controller.registerHandler(DELETE, "/{index}/_warmer/{name}", this); - controller.registerHandler(DELETE, "/{index}/{type}/_warmer/{name}", this); + controller.registerHandler(DELETE, "/{index}/_warmers", this); + controller.registerHandler(DELETE, "/{index}/_warmers/{name}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { - DeleteWarmerRequest deleteWarmerRequest = new DeleteWarmerRequest(request.param("name")) + DeleteWarmerRequest deleteWarmerRequest = new DeleteWarmerRequest(Strings.splitStringByCommaToArray(request.param("name"))) .indices(Strings.splitStringByCommaToArray(request.param("index"))); deleteWarmerRequest.listenerThreaded(false); deleteWarmerRequest.timeout(request.paramAsTime("timeout", deleteWarmerRequest.timeout())); diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java index 4a1d03435e0..b0bc3b1f35e 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/put/RestPutWarmerAction.java @@ -27,6 +27,7 @@ import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.rest.*; +import static org.elasticsearch.rest.RestRequest.Method.POST; import static org.elasticsearch.rest.RestRequest.Method.PUT; /** @@ -36,8 +37,21 @@ public class RestPutWarmerAction extends BaseRestHandler { @Inject public RestPutWarmerAction(Settings settings, Client client, RestController controller) { super(settings, client); + controller.registerHandler(PUT, "/_warmer/{name}", this); controller.registerHandler(PUT, "/{index}/_warmer/{name}", this); controller.registerHandler(PUT, "/{index}/{type}/_warmer/{name}", this); + + controller.registerHandler(PUT, "/_warmers/{name}", this); + controller.registerHandler(PUT, "/{index}/_warmers/{name}", this); + controller.registerHandler(PUT, "/{index}/{type}/_warmers/{name}", this); + + controller.registerHandler(POST, "/_warmer/{name}", this); + controller.registerHandler(POST, "/{index}/_warmer/{name}", this); + controller.registerHandler(POST, "/{index}/{type}/_warmer/{name}", this); + + controller.registerHandler(POST, "/_warmers/{name}", this); + controller.registerHandler(POST, "/{index}/_warmers/{name}", this); + controller.registerHandler(POST, "/{index}/{type}/_warmers/{name}", this); } @Override diff --git a/src/main/java/org/elasticsearch/search/warmer/IndexWarmerMissingException.java b/src/main/java/org/elasticsearch/search/warmer/IndexWarmerMissingException.java index 7e8517e4b08..3892714bb0d 100644 --- a/src/main/java/org/elasticsearch/search/warmer/IndexWarmerMissingException.java +++ b/src/main/java/org/elasticsearch/search/warmer/IndexWarmerMissingException.java @@ -21,20 +21,22 @@ package org.elasticsearch.search.warmer; import org.elasticsearch.ElasticsearchException; import org.elasticsearch.rest.RestStatus; +import java.util.Arrays; + /** * */ public class IndexWarmerMissingException extends ElasticsearchException { - private final String name; + private final String[] names; - public IndexWarmerMissingException(String name) { - super("index_warmer [" + name + "] missing"); - this.name = name; + public IndexWarmerMissingException(String... names) { + super("index_warmer [" + Arrays.toString(names) + "] missing"); + this.names = names; } - public String name() { - return this.name; + public String[] names() { + return this.names; } diff --git a/src/test/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequestTests.java b/src/test/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequestTests.java index 3cb7e1e5798..386abbd3efb 100644 --- a/src/test/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequestTests.java +++ b/src/test/java/org/elasticsearch/action/admin/indices/warmer/delete/DeleteWarmerRequestTests.java @@ -48,7 +48,7 @@ public class DeleteWarmerRequestTests extends ElasticsearchTestCase { DeleteWarmerRequest inRequest = new DeleteWarmerRequest(); inRequest.readFrom(esBuffer); - assertThat(inRequest.name(), equalTo("warmer1")); + assertThat(inRequest.names()[0], equalTo("warmer1")); //timeout is default as we don't read it from the received buffer assertThat(inRequest.timeout().millis(), equalTo(new DeleteWarmerRequest().timeout().millis())); @@ -70,7 +70,7 @@ public class DeleteWarmerRequestTests extends ElasticsearchTestCase { DeleteWarmerRequest inRequest = new DeleteWarmerRequest(); inRequest.readFrom(esBuffer); - assertThat(inRequest.name(), equalTo("warmer1")); + assertThat(inRequest.names()[0], equalTo("warmer1")); //timeout is default as we don't read it from the received buffer assertThat(inRequest.timeout().millis(), equalTo(outRequest.timeout().millis())); diff --git a/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java b/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java index 5599857fd18..c01abde04fe 100644 --- a/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java +++ b/src/test/java/org/elasticsearch/aliases/IndexAliasesTests.java @@ -35,6 +35,8 @@ import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.index.query.FilterBuilder; import org.elasticsearch.index.query.QueryBuilders; +import org.elasticsearch.indices.IndexMissingException; +import org.elasticsearch.rest.action.admin.indices.alias.delete.AliasesMissingException; import org.elasticsearch.search.SearchHit; import org.elasticsearch.search.SearchHits; import org.elasticsearch.search.facet.FacetBuilders; @@ -90,7 +92,7 @@ public class IndexAliasesTests extends ElasticsearchIntegrationTest { ensureGreen(); logger.info("--> remove [alias1], Aliasing index [test_x] with [alias1]"); - admin().indices().aliases(indexAliasesRequest().removeAlias("test", "alias1").addAlias("test_x", "alias1")).actionGet(); + admin().indices().aliases(indexAliasesRequest().removeAlias("test", "alias1").addAlias("alias1", "test_x")).actionGet(); Thread.sleep(300); logger.info("--> indexing against [alias1], should work against [test_x]"); @@ -412,6 +414,43 @@ public class IndexAliasesTests extends ElasticsearchIntegrationTest { assertHits(searchResponse.getHits(), "4"); } + + + @Test + public void testDeleteAliases() throws Exception { + + logger.info("--> creating index [test1]"); + admin().indices().create(createIndexRequest("test1")).actionGet(); + + logger.info("--> creating index [test2]"); + admin().indices().create(createIndexRequest("test2")).actionGet(); + + ensureGreen(); + + logger.info("--> adding filtering aliases to index [test1]"); + admin().indices().prepareAliases().addAlias("test1", "aliasToTest1").execute().actionGet(); + admin().indices().prepareAliases().addAlias("test1", "aliasToTests").execute().actionGet(); + admin().indices().prepareAliases().addAlias("test1", "foos", termFilter("name", "foo")).execute().actionGet(); + admin().indices().prepareAliases().addAlias("test1", "bars", termFilter("name", "bar")).execute().actionGet(); + admin().indices().prepareAliases().addAlias("test1", "tests", termFilter("name", "test")).execute().actionGet(); + + logger.info("--> adding filtering aliases to index [test2]"); + admin().indices().prepareAliases().addAlias("test2", "aliasToTest2").execute().actionGet(); + admin().indices().prepareAliases().addAlias("test2", "aliasToTests").execute().actionGet(); + admin().indices().prepareAliases().addAlias("test2", "foos", termFilter("name", "foo")).execute().actionGet(); + admin().indices().prepareAliases().addAlias("test2", "tests", termFilter("name", "test")).execute().actionGet(); + + String[] indices = {"test1", "test2"}; + String[] aliases = {"aliasToTest1", "foos", "bars", "tests", "aliasToTest2", "aliasToTests"}; + + admin().indices().prepareAliases().removeAlias(indices, aliases).execute().actionGet(); + + AliasesExistResponse response = admin().indices().prepareAliasesExist(aliases).execute().actionGet(); + assertThat(response.exists(), equalTo(false)); + + } + + @Test public void testWaitForAliasCreationMultipleShards() throws Exception { @@ -512,10 +551,16 @@ public class IndexAliasesTests extends ElasticsearchIntegrationTest { assertThat(admin().indices().prepareAliases().removeAlias("test", "alias1").setTimeout(timeout).execute().actionGet().isAcknowledged(), equalTo(true)); assertThat(stopWatch.stop().lastTaskTime().millis(), lessThan(timeout.millis())); - logger.info("--> deleting alias1 one more time"); - stopWatch.start(); - assertThat(admin().indices().prepareAliases().removeAlias("test", "alias1").setTimeout(timeout).execute().actionGet().isAcknowledged(), equalTo(true)); - assertThat(stopWatch.stop().lastTaskTime().millis(), lessThan(timeout.millis())); + + } + + @Test(expected = AliasesMissingException.class) + public void testIndicesRemoveNonExistingAliasResponds404() throws Exception { + logger.info("--> creating index [test]"); + createIndex("test"); + ensureGreen(); + logger.info("--> deleting alias1 which does not exist"); + assertThat(admin().indices().prepareAliases().removeAlias("test", "alias1").execute().actionGet().isAcknowledged(), equalTo(true)); } @Test @@ -729,7 +774,7 @@ public class IndexAliasesTests extends ElasticsearchIntegrationTest { assertThat(existsResponse.exists(), equalTo(false)); } - @Test(expected = ActionRequestValidationException.class) + @Test(expected = IndexMissingException.class) public void testAddAliasNullIndex() { admin().indices().prepareAliases().addAliasAction(AliasAction.newAddAliasAction(null, "alias1")) @@ -765,7 +810,7 @@ public class IndexAliasesTests extends ElasticsearchIntegrationTest { assertTrue("Should throw " + ActionRequestValidationException.class.getSimpleName(), false); } catch (ActionRequestValidationException e) { assertThat(e.validationErrors(), notNullValue()); - assertThat(e.validationErrors().size(), equalTo(2)); + assertThat(e.validationErrors().size(), equalTo(1)); } } @@ -808,7 +853,7 @@ public class IndexAliasesTests extends ElasticsearchIntegrationTest { @Test public void testRemoveAliasNullAliasNullIndex() { try { - admin().indices().prepareAliases().addAliasAction(AliasAction.newAddAliasAction(null, null)) + admin().indices().prepareAliases().addAliasAction(AliasAction.newRemoveAliasAction(null, null)) .execute().actionGet(); assertTrue("Should throw " + ActionRequestValidationException.class.getSimpleName(), false); } catch (ActionRequestValidationException e) { diff --git a/src/test/java/org/elasticsearch/cluster/ack/AckTests.java b/src/test/java/org/elasticsearch/cluster/ack/AckTests.java index c69964fe8cc..15bb212977c 100644 --- a/src/test/java/org/elasticsearch/cluster/ack/AckTests.java +++ b/src/test/java/org/elasticsearch/cluster/ack/AckTests.java @@ -123,7 +123,7 @@ public class AckTests extends ElasticsearchIntegrationTest { assertAcked(client().admin().indices().preparePutWarmer("custom_warmer") .setSearchRequest(client().prepareSearch("test").setTypes("test").setQuery(QueryBuilders.matchAllQuery()))); - assertAcked(client().admin().indices().prepareDeleteWarmer().setIndices("test").setName("custom_warmer")); + assertAcked(client().admin().indices().prepareDeleteWarmer().setIndices("test").setNames("custom_warmer")); for (Client client : clients()) { GetWarmersResponse getWarmersResponse = client.admin().indices().prepareGetWarmers().setLocal(true).get(); diff --git a/src/test/java/org/elasticsearch/cluster/metadata/MetaDataTests.java b/src/test/java/org/elasticsearch/cluster/metadata/MetaDataTests.java index 12b6a4fb323..8ff8c26a13f 100644 --- a/src/test/java/org/elasticsearch/cluster/metadata/MetaDataTests.java +++ b/src/test/java/org/elasticsearch/cluster/metadata/MetaDataTests.java @@ -369,37 +369,37 @@ public class MetaDataTests extends ElasticsearchTestCase { @Test public void testIsExplicitAllIndices_null() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(null), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(null), equalTo(false)); } @Test public void testIsExplicitAllIndices_empty() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[0]), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(new String[0]), equalTo(false)); } @Test public void testIsExplicitAllIndices_explicitAll() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[]{"_all"}), equalTo(true)); + assertThat(metaData.isExplicitAllPattern(new String[]{"_all"}), equalTo(true)); } @Test public void testIsExplicitAllIndices_explicitAllPlusOther() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[]{"_all", "other"}), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(new String[]{"_all", "other"}), equalTo(false)); } @Test public void testIsExplicitAllIndices_normalIndexes() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[]{"index1", "index2", "index3"}), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(new String[]{"index1", "index2", "index3"}), equalTo(false)); } @Test public void testIsExplicitAllIndices_wildcard() throws Exception { MetaData metaData = MetaData.builder().build(); - assertThat(metaData.isExplicitAllIndices(new String[]{"*"}), equalTo(false)); + assertThat(metaData.isExplicitAllPattern(new String[]{"*"}), equalTo(false)); } @Test diff --git a/src/test/java/org/elasticsearch/indices/IndicesOptionsTests.java b/src/test/java/org/elasticsearch/indices/IndicesOptionsTests.java index e0e19a62783..ff1b391657a 100644 --- a/src/test/java/org/elasticsearch/indices/IndicesOptionsTests.java +++ b/src/test/java/org/elasticsearch/indices/IndicesOptionsTests.java @@ -55,6 +55,7 @@ import org.elasticsearch.cluster.block.ClusterBlockException; import org.elasticsearch.common.Strings; import org.elasticsearch.common.bytes.BytesArray; import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.index.query.QueryBuilders; import org.elasticsearch.search.suggest.SuggestBuilder; import org.elasticsearch.search.warmer.IndexWarmersMetaData; import org.elasticsearch.test.ElasticsearchIntegrationTest; @@ -537,6 +538,103 @@ public class IndicesOptionsTests extends ElasticsearchIntegrationTest { assertThat(client().admin().indices().prepareTypesExists("barbaz").setTypes("type1").get().isExists(), equalTo(false)); } + + @Test + public void testPutWarmer() throws Exception { + assertAcked(prepareCreate("foobar")); + ensureYellow(); + verify(client().admin().indices().preparePutWarmer("warmer1").setSearchRequest(client().prepareSearch().setIndices("foobar").setQuery(QueryBuilders.matchAllQuery())), false); + assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("warmer1").get().getWarmers().size(), equalTo(1)); + + } + + @Test + public void testPutWarmer_wildcard() throws Exception { + + assertAcked(prepareCreate("foo")); + assertAcked(prepareCreate("foobar")); + assertAcked(prepareCreate("bar")); + assertAcked(prepareCreate("barbaz")); + ensureYellow(); + + verify(client().admin().indices().preparePutWarmer("warmer1").setSearchRequest(client().prepareSearch().setIndices("foo*").setQuery(QueryBuilders.matchAllQuery())), false); + + assertThat(client().admin().indices().prepareGetWarmers("foo").setWarmers("warmer1").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("warmer1").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("bar").setWarmers("warmer1").get().getWarmers().size(), equalTo(0)); + assertThat(client().admin().indices().prepareGetWarmers("barbaz").setWarmers("warmer1").get().getWarmers().size(), equalTo(0)); + + verify(client().admin().indices().preparePutWarmer("warmer2").setSearchRequest(client().prepareSearch().setIndices().setQuery(QueryBuilders.matchAllQuery())), false); + + assertThat(client().admin().indices().prepareGetWarmers("foo").setWarmers("warmer2").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("warmer2").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("bar").setWarmers("warmer2").get().getWarmers().size(), equalTo(1)); + assertThat(client().admin().indices().prepareGetWarmers("barbaz").setWarmers("warmer2").get().getWarmers().size(), equalTo(1)); + + } + + @Test + public void testPutAlias() throws Exception { + assertAcked(prepareCreate("foobar")); + ensureYellow(); + verify(client().admin().indices().prepareAliases().addAlias("foobar", "foobar_alias"), false); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foobar").get().exists(), equalTo(true)); + + } + + @Test + public void testPutAlias_wildcard() throws Exception { + + assertAcked(prepareCreate("foo")); + assertAcked(prepareCreate("foobar")); + assertAcked(prepareCreate("bar")); + assertAcked(prepareCreate("barbaz")); + ensureYellow(); + + verify(client().admin().indices().prepareAliases().addAlias("foo*", "foobar_alias"), false); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foo").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foobar").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("bar").get().exists(), equalTo(false)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("barbaz").get().exists(), equalTo(false)); + + verify(client().admin().indices().prepareAliases().addAlias("*", "foobar_alias"), false); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foo").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("foobar").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("bar").get().exists(), equalTo(true)); + assertThat(client().admin().indices().prepareAliasesExist("foobar_alias").setIndices("barbaz").get().exists(), equalTo(true)); + + } + @Test + public void testDeleteMapping_typeWildcard() throws Exception { + verify(client().admin().indices().prepareDeleteMapping("_all").setType("type1"), true); + + assertAcked(prepareCreate("foo").addMapping("type1", "field", "type=string")); + assertAcked(prepareCreate("foobar").addMapping("type2", "field", "type=string")); + assertAcked(prepareCreate("bar").addMapping("type3", "field", "type=string")); + assertAcked(prepareCreate("barbaz").addMapping("type4", "field", "type=string")); + + ensureYellow(); + + assertThat(client().admin().indices().prepareTypesExists("foo").setTypes("type1").get().isExists(), equalTo(true)); + assertThat(client().admin().indices().prepareTypesExists("foobar").setTypes("type2").get().isExists(), equalTo(true)); + assertThat(client().admin().indices().prepareTypesExists("bar").setTypes("type3").get().isExists(), equalTo(true)); + assertThat(client().admin().indices().prepareTypesExists("barbaz").setTypes("type4").get().isExists(), equalTo(true)); + + verify(client().admin().indices().prepareDeleteMapping("foo*").setType("type*"), false); + assertThat(client().admin().indices().prepareTypesExists("foo").setTypes("type1").get().isExists(), equalTo(false)); + assertThat(client().admin().indices().prepareTypesExists("foobar").setTypes("type2").get().isExists(), equalTo(false)); + assertThat(client().admin().indices().prepareTypesExists("bar").setTypes("type3").get().isExists(), equalTo(true)); + assertThat(client().admin().indices().prepareTypesExists("barbaz").setTypes("type4").get().isExists(), equalTo(true)); + + assertAcked(client().admin().indices().prepareDelete("foo*")); + + verify(client().admin().indices().prepareDeleteMapping("foo*").setType("type1"), true); + + verify(client().admin().indices().prepareDeleteMapping("_all").setType("type3","type4"), false); + assertThat(client().admin().indices().prepareTypesExists("bar").setTypes("type3").get().isExists(), equalTo(false)); + assertThat(client().admin().indices().prepareTypesExists("barbaz").setTypes("type4").get().isExists(), equalTo(false)); + } + @Test public void testDeleteWarmer() throws Exception { IndexWarmersMetaData.Entry entry = new IndexWarmersMetaData.Entry( @@ -545,15 +643,15 @@ public class IndicesOptionsTests extends ElasticsearchIntegrationTest { assertAcked(prepareCreate("foobar").addCustom(new IndexWarmersMetaData(entry))); ensureYellow(); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo").setName("test1"), true); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo").setNames("test1"), true); assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("test1").get().getWarmers().size(), equalTo(1)); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("foobar").setName("test1"), false); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("foobar").setNames("test1"), false); assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("test1").get().getWarmers().size(), equalTo(0)); } @Test public void testDeleteWarmer_wildcard() throws Exception { - verify(client().admin().indices().prepareDeleteWarmer().setIndices("_all").setName("test1"), true); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("_all").setNames("test1"), true); IndexWarmersMetaData.Entry entry = new IndexWarmersMetaData.Entry( "test1", new String[]{"type1"}, new BytesArray("{\"query\" : { \"match_all\" : {}}}") @@ -564,7 +662,7 @@ public class IndicesOptionsTests extends ElasticsearchIntegrationTest { assertAcked(prepareCreate("barbaz").addCustom(new IndexWarmersMetaData(entry))); ensureYellow(); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo*").setName("test1"), false); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo*").setNames("test1"), false); assertThat(client().admin().indices().prepareGetWarmers("foo").setWarmers("test1").get().getWarmers().size(), equalTo(0)); assertThat(client().admin().indices().prepareGetWarmers("foobar").setWarmers("test1").get().getWarmers().size(), equalTo(0)); assertThat(client().admin().indices().prepareGetWarmers("bar").setWarmers("test1").get().getWarmers().size(), equalTo(1)); @@ -572,9 +670,9 @@ public class IndicesOptionsTests extends ElasticsearchIntegrationTest { assertAcked(client().admin().indices().prepareDelete("foo*")); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo*").setName("test1"), true); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("foo*").setNames("test1"), true); - verify(client().admin().indices().prepareDeleteWarmer().setIndices("_all").setName("test1"), false); + verify(client().admin().indices().prepareDeleteWarmer().setIndices("_all").setNames("test1"), false); assertThat(client().admin().indices().prepareGetWarmers("bar").setWarmers("test1").get().getWarmers().size(), equalTo(0)); assertThat(client().admin().indices().prepareGetWarmers("barbaz").setWarmers("test1").get().getWarmers().size(), equalTo(0)); } @@ -622,12 +720,18 @@ public class IndicesOptionsTests extends ElasticsearchIntegrationTest { assertThat(client().admin().indices().prepareGetMappings("foobar").get().mappings().get("foobar").get("type2"), notNullValue()); assertThat(client().admin().indices().prepareGetMappings("bar").get().mappings().get("bar").get("type2"), notNullValue()); assertThat(client().admin().indices().prepareGetMappings("barbaz").get().mappings().get("barbaz").get("type2"), notNullValue()); + verify(client().admin().indices().preparePutMapping().setType("type3").setSource("field", "type=string"), false); + assertThat(client().admin().indices().prepareGetMappings("foo").get().mappings().get("foo").get("type3"), notNullValue()); + assertThat(client().admin().indices().prepareGetMappings("foobar").get().mappings().get("foobar").get("type3"), notNullValue()); + assertThat(client().admin().indices().prepareGetMappings("bar").get().mappings().get("bar").get("type3"), notNullValue()); + assertThat(client().admin().indices().prepareGetMappings("barbaz").get().mappings().get("barbaz").get("type3"), notNullValue()); + verify(client().admin().indices().preparePutMapping("c*").setType("type1").setSource("field", "type=string"), true); assertAcked(client().admin().indices().prepareClose("barbaz").get()); - verify(client().admin().indices().preparePutMapping("barbaz").setType("type3").setSource("field", "type=string"), false); - assertThat(client().admin().indices().prepareGetMappings("barbaz").get().mappings().get("barbaz").get("type3"), notNullValue()); + verify(client().admin().indices().preparePutMapping("barbaz").setType("type4").setSource("field", "type=string"), false); + assertThat(client().admin().indices().prepareGetMappings("barbaz").get().mappings().get("barbaz").get("type4"), notNullValue()); } @Test diff --git a/src/test/java/org/elasticsearch/indices/mapping/SimpleDeleteMappingTests.java b/src/test/java/org/elasticsearch/indices/mapping/SimpleDeleteMappingTests.java index a35f5139f99..3499b88238d 100644 --- a/src/test/java/org/elasticsearch/indices/mapping/SimpleDeleteMappingTests.java +++ b/src/test/java/org/elasticsearch/indices/mapping/SimpleDeleteMappingTests.java @@ -19,14 +19,17 @@ package org.elasticsearch.indices.mapping; +import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse; import org.elasticsearch.action.count.CountResponse; import org.elasticsearch.cluster.ClusterState; import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.elasticsearch.test.hamcrest.ElasticsearchAssertions; import org.junit.Test; import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; import static org.hamcrest.Matchers.*; /** @@ -43,7 +46,7 @@ public class SimpleDeleteMappingTests extends ElasticsearchIntegrationTest { } ensureGreen(); - client().admin().indices().prepareRefresh().execute().actionGet(); + refresh(); for (int i = 0; i < 10; i++) { CountResponse countResponse = client().prepareCount().setQuery(matchAllQuery()).execute().actionGet(); @@ -51,15 +54,13 @@ public class SimpleDeleteMappingTests extends ElasticsearchIntegrationTest { } ClusterState clusterState = client().admin().cluster().prepareState().execute().actionGet().getState(); - for (int i = 0; i < 10 && !clusterState.metaData().index("test").mappings().containsKey("type1"); i++, Thread.sleep(100)) ; assertThat(clusterState.metaData().index("test").mappings().containsKey("type1"), equalTo(true)); GetMappingsResponse mappingsResponse = client().admin().indices().prepareGetMappings("test").setTypes("type1").execute().actionGet(); assertThat(mappingsResponse.getMappings().get("test").get("type1"), notNullValue()); - client().admin().indices().prepareDeleteMapping().setType("type1").execute().actionGet(); - Thread.sleep(500); // for now, we don't have ack logic, so just wait + ElasticsearchAssertions.assertAcked(client().admin().indices().prepareDeleteMapping().setIndices("test").setType("type1")); for (int i = 0; i < 10; i++) { CountResponse countResponse = client().prepareCount().setQuery(matchAllQuery()).execute().actionGet(); @@ -71,4 +72,35 @@ public class SimpleDeleteMappingTests extends ElasticsearchIntegrationTest { mappingsResponse = client().admin().indices().prepareGetMappings("test").setTypes("type1").execute().actionGet(); assertThat(mappingsResponse.getMappings().get("test"), nullValue()); } + + + @Test + public void deleteMappingAllowNoBlankIndexAndNoEmptyStrings() throws Exception { + assertAcked(client().admin().indices().prepareCreate("index1").addMapping("1", "field1", "type=string").get()); + assertAcked(client().admin().indices().prepareCreate("1index").addMapping("1", "field1", "type=string").get()); + + // Should succeed, since no wildcards + client().admin().indices().prepareDeleteMapping("1index").setType("1").get(); + try { + client().admin().indices().prepareDeleteMapping("_all").get(); + fail(); + } catch (ActionRequestValidationException e) {} + + try { + client().admin().indices().prepareDeleteMapping("_all").setType("").get(); + fail(); + } catch (ActionRequestValidationException e) {} + + try { + client().admin().indices().prepareDeleteMapping().setType("1").get(); + fail(); + } catch (ActionRequestValidationException e) {} + + try { + client().admin().indices().prepareDeleteMapping("").setType("1").get(); + fail(); + } catch (ActionRequestValidationException e) {} + + } + } diff --git a/src/test/java/org/elasticsearch/indices/warmer/LocalGatewayIndicesWarmerTests.java b/src/test/java/org/elasticsearch/indices/warmer/LocalGatewayIndicesWarmerTests.java index 080d0f9bff4..5a056c5e0ee 100644 --- a/src/test/java/org/elasticsearch/indices/warmer/LocalGatewayIndicesWarmerTests.java +++ b/src/test/java/org/elasticsearch/indices/warmer/LocalGatewayIndicesWarmerTests.java @@ -127,7 +127,7 @@ public class LocalGatewayIndicesWarmerTests extends ElasticsearchIntegrationTest logger.info("--> delete warmer warmer_1"); - DeleteWarmerResponse deleteWarmerResponse = client().admin().indices().prepareDeleteWarmer().setIndices("test").setName("warmer_1").execute().actionGet(); + DeleteWarmerResponse deleteWarmerResponse = client().admin().indices().prepareDeleteWarmer().setIndices("test").setNames("warmer_1").execute().actionGet(); assertThat(deleteWarmerResponse.isAcknowledged(), equalTo(true)); logger.info("--> verify warmers (delete) are registered in cluster state"); diff --git a/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java b/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java index 5e8d429767a..e1bcac1d2a9 100644 --- a/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java +++ b/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java @@ -175,10 +175,10 @@ public class SimpleIndicesWarmerTests extends ElasticsearchIntegrationTest { createIndex("test"); try { - client().admin().indices().prepareDeleteWarmer().setIndices("test").setName("foo").execute().actionGet(1000); + client().admin().indices().prepareDeleteWarmer().setIndices("test").setNames("foo").execute().actionGet(1000); assert false : "warmer foo should not exist"; } catch (IndexWarmerMissingException ex) { - assertThat(ex.name(), equalTo("foo")); + assertThat(ex.names()[0], equalTo("foo")); } } @@ -199,7 +199,7 @@ public class SimpleIndicesWarmerTests extends ElasticsearchIntegrationTest { assertThat(entry.value.size(), equalTo(1)); assertThat(entry.value.iterator().next().name(), equalTo("custom_warmer")); - DeleteWarmerResponse deleteWarmerResponse = client().admin().indices().prepareDeleteWarmer().setIndices("test").setName("custom_warmer").get(); + DeleteWarmerResponse deleteWarmerResponse = client().admin().indices().prepareDeleteWarmer().setIndices("test").setNames("custom_warmer").get(); assertThat(deleteWarmerResponse.isAcknowledged(), equalTo(true)); getWarmersResponse = client().admin().indices().prepareGetWarmers("test").get(); diff --git a/src/test/java/org/elasticsearch/mlt/MoreLikeThisActionTests.java b/src/test/java/org/elasticsearch/mlt/MoreLikeThisActionTests.java index 5b0489851a1..40ad9ee42d7 100644 --- a/src/test/java/org/elasticsearch/mlt/MoreLikeThisActionTests.java +++ b/src/test/java/org/elasticsearch/mlt/MoreLikeThisActionTests.java @@ -93,8 +93,8 @@ public class MoreLikeThisActionTests extends ElasticsearchIntegrationTest { .startObject("text").field("type", "string").endObject() .endObject().endObject().endObject())); logger.info("Creating aliases alias release"); - client().admin().indices().aliases(indexAliasesRequest().addAlias("test", "release", termFilter("text", "release"))).actionGet(); - client().admin().indices().aliases(indexAliasesRequest().addAlias("test", "beta", termFilter("text", "beta"))).actionGet(); + client().admin().indices().aliases(indexAliasesRequest().addAlias("release", termFilter("text", "release"), "test")).actionGet(); + client().admin().indices().aliases(indexAliasesRequest().addAlias("beta", termFilter("text", "beta"), "test")).actionGet(); logger.info("Running Cluster Health"); assertThat(ensureGreen(), equalTo(ClusterHealthStatus.GREEN)); diff --git a/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java b/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java index cd0ab6efec8..d752bdea3e2 100644 --- a/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java +++ b/src/test/java/org/elasticsearch/operateAllIndices/DestructiveOperationsIntegrationTests.java @@ -149,7 +149,8 @@ public class DestructiveOperationsIntegrationTests extends ElasticsearchIntegrat .put(DestructiveOperations.REQUIRES_NAME, true) .build(); assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(settings)); - + + assertAcked(client().admin().indices().prepareCreate("index1").addMapping("1", "field1", "type=string").get()); assertAcked(client().admin().indices().prepareCreate("1index").addMapping("1", "field1", "type=string").get()); @@ -158,20 +159,19 @@ public class DestructiveOperationsIntegrationTests extends ElasticsearchIntegrat try { client().admin().indices().prepareDeleteMapping("_all").setType("1").get(); fail(); - } catch (ElasticsearchIllegalArgumentException e) {} - + } catch (ElasticsearchIllegalArgumentException e) { + } try { - client().admin().indices().prepareDeleteMapping().setType("1").get(); + client().admin().indices().prepareDeleteMapping().setIndices("*").setType("1").get(); fail(); - } catch (ElasticsearchIllegalArgumentException e) {} + } catch (ElasticsearchIllegalArgumentException e) { + } - settings = ImmutableSettings.builder() - .put(DestructiveOperations.REQUIRES_NAME, false) - .build(); + settings = ImmutableSettings.builder().put(DestructiveOperations.REQUIRES_NAME, false).build(); assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(settings)); assertAcked(client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string")); - assertAcked(client().admin().indices().prepareDeleteMapping().setType("1")); + assertAcked(client().admin().indices().prepareDeleteMapping().setIndices("*").setType("1")); assertAcked(client().admin().indices().preparePutMapping("1index").setType("1").setSource("field1", "type=string")); assertAcked(client().admin().indices().prepareDeleteMapping("_all").setType("1")); } From fb3ea1feb0951c0f18a9227826ce99327a443f17 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 14 Jan 2014 20:38:03 +0100 Subject: [PATCH 14/34] Use assertThat instead of plain asserts in MemoryCircuitBreakerTests --- .../common/breaker/MemoryCircuitBreakerTests.java | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/test/java/org/elasticsearch/common/breaker/MemoryCircuitBreakerTests.java b/src/test/java/org/elasticsearch/common/breaker/MemoryCircuitBreakerTests.java index 7a07d0b03b6..5565cd44c0d 100644 --- a/src/test/java/org/elasticsearch/common/breaker/MemoryCircuitBreakerTests.java +++ b/src/test/java/org/elasticsearch/common/breaker/MemoryCircuitBreakerTests.java @@ -54,7 +54,7 @@ public class MemoryCircuitBreakerTests extends ElasticsearchTestCase { if (tripped.get()) { assertThat("tripped too many times", true, equalTo(false)); } else { - assert tripped.compareAndSet(false, true); + assertThat(tripped.compareAndSet(false, true), equalTo(true)); } } catch (Throwable e2) { lastException.set(e2); @@ -86,7 +86,6 @@ public class MemoryCircuitBreakerTests extends ElasticsearchTestCase { breaker.addEstimateBytesAndMaybeBreak(3); fail("should never reach this"); } catch (CircuitBreakingException cbe) { - assert true; } // shouldn't throw an exception @@ -102,7 +101,6 @@ public class MemoryCircuitBreakerTests extends ElasticsearchTestCase { breaker.addEstimateBytesAndMaybeBreak(0); fail("should never reach this"); } catch (CircuitBreakingException cbe) { - assert true; } } } From ba7699a38b1bdaaccef20fb28888dd5fe5861ace Mon Sep 17 00:00:00 2001 From: Igor Motov Date: Tue, 14 Jan 2014 16:20:46 -0500 Subject: [PATCH 15/34] Add documentation for index.routing.allocation.*._name and index.routing.allocation.*._id options --- docs/reference/modules/cluster.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/modules/cluster.asciidoc b/docs/reference/modules/cluster.asciidoc index f089e2dad90..9d0eefe3076 100644 --- a/docs/reference/modules/cluster.asciidoc +++ b/docs/reference/modules/cluster.asciidoc @@ -206,7 +206,8 @@ The `include`, `exclude` and `require` values can have generic simple matching wildcards, for example, `value1*`. A special attribute name called `_ip` can be used to match on node ip values. In addition `_host` attribute can be used to match on either the node's hostname or its ip -address. +address. Similarly `_name` and `_id` attributes can be used to match on +node name and node id accordingly. Obviously a node can have several attributes associated with it, and both the attribute name and value are controlled in the setting. For From 158483554dc2c11a0168ffbb728bece2619fec21 Mon Sep 17 00:00:00 2001 From: Andrew Raines Date: Mon, 13 Jan 2014 19:35:13 -0600 Subject: [PATCH 16/34] Shorten epoch to second precision. Closes #4696. --- .../org/elasticsearch/rest/action/cat/RestCountAction.java | 4 ++-- .../org/elasticsearch/rest/action/cat/RestHealthAction.java | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java index 76181761e9e..7cb754bf579 100644 --- a/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java +++ b/src/main/java/org/elasticsearch/rest/action/cat/RestCountAction.java @@ -98,7 +98,7 @@ public class RestCountAction extends AbstractCatAction { Table getTableWithHeader(final RestRequest request) { Table table = new Table(); table.startHeaders(); - table.addCell("time(ms)", "desc:time, in milliseconds since epoch UTC, that the count was executed"); + table.addCell("epoch", "desc:seconds since 1970-01-01 00:00:00, that the count was executed"); table.addCell("timestamp", "desc:time that the count was executed"); table.addCell("count", "desc:the document count"); table.endHeaders(); @@ -109,7 +109,7 @@ public class RestCountAction extends AbstractCatAction { private Table buildTable(RestRequest request, CountResponse response) { Table table = getTableWithHeader(request); - long time = System.currentTimeMillis(); + long time = System.currentTimeMillis() / 1000; table.startRow(); table.addCell(time); table.addCell(dateFormat.print(time)); diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java index 83a68f486c1..d3f0a96bd68 100644 --- a/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java +++ b/src/main/java/org/elasticsearch/rest/action/cat/RestHealthAction.java @@ -81,7 +81,7 @@ public class RestHealthAction extends AbstractCatAction { Table getTableWithHeader(final RestRequest request) { Table t = new Table(); t.startHeaders(); - t.addCell("time(ms)", "desc:time, in milliseconds since epoch UTC, that the count was executed"); + t.addCell("epoch", "desc:seconds since 1970-01-01 00:00:00, that the count was executed"); t.addCell("timestamp", "desc:time that the count was executed"); t.addCell("cluster", "desc:cluster name"); t.addCell("status", "desc:health status"); @@ -100,7 +100,7 @@ public class RestHealthAction extends AbstractCatAction { private DateTimeFormatter dateFormat = DateTimeFormat.forPattern("HH:mm:ss"); private Table buildTable(final ClusterHealthResponse health, final RestRequest request) { - long time = System.currentTimeMillis(); + long time = System.currentTimeMillis() / 1000; Table t = getTableWithHeader(request); t.startRow(); t.addCell(time); From b35ca1aa751efc551f2e33b9d21be27b5e8dd324 Mon Sep 17 00:00:00 2001 From: Andrew Raines Date: Mon, 13 Jan 2014 21:00:56 -0600 Subject: [PATCH 17/34] Fix disk percent used calc in cat/allocation Closes #4670. --- .../rest/action/cat/RestAllocationAction.java | 27 +++++++------------ 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java b/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java index 29c88256d81..bd3f88d3891 100644 --- a/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java +++ b/src/main/java/org/elasticsearch/rest/action/cat/RestAllocationAction.java @@ -117,7 +117,8 @@ public class RestAllocationAction extends AbstractCatAction { table.addCell("shards", "text-align:right;desc:number of shards on node"); table.addCell("diskUsed", "text-align:right;desc:disk used (total, not just ES)"); table.addCell("diskAvail", "text-align:right;desc:disk available"); - table.addCell("diskRatio", "text-align:right;desc:percent disk used"); + table.addCell("diskTotal", "text-align:right;desc:total capacity of all volumes"); + table.addCell("diskPercent", "text-align:right;desc:percent disk used"); table.addCell("host", "desc:host of node"); table.addCell("ip", "desc:ip of node"); table.addCell("node", "desc:name of node"); @@ -143,34 +144,25 @@ public class RestAllocationAction extends AbstractCatAction { for (NodeStats nodeStats : stats.getNodes()) { DiscoveryNode node = nodeStats.getNode(); - long used = -1; - long avail = -1; - - Iterator diskIter = nodeStats.getFs().iterator(); - while (diskIter.hasNext()) { - FsStats.Info disk = diskIter.next(); - used += disk.getTotal().bytes() - disk.getAvailable().bytes(); - avail += disk.getAvailable().bytes(); - } - - String nodeId = node.id(); - int shardCount = -1; - if (allocs.containsKey(nodeId)) { + if (allocs.containsKey(node.id())) { shardCount = allocs.lget(); } - float ratio = -1; + long used = nodeStats.getFs().getTotal().getTotal().bytes() - nodeStats.getFs().getTotal().getAvailable().bytes(); + long avail = nodeStats.getFs().getTotal().getAvailable().bytes(); + short diskPercent = -1; if (used >= 0 && avail > 0) { - ratio = used / (float) avail; + diskPercent = (short) (used * 100 / (used + avail)); } table.startRow(); table.addCell(shardCount < 0 ? null : shardCount); table.addCell(used < 0 ? null : new ByteSizeValue(used)); table.addCell(avail < 0 ? null : new ByteSizeValue(avail)); - table.addCell(ratio < 0 ? null : String.format(Locale.ROOT, "%.1f%%", ratio * 100.0)); + table.addCell(nodeStats.getFs().getTotal().getTotal()); + table.addCell(diskPercent < 0 ? null : diskPercent); table.addCell(node == null ? null : node.getHostName()); table.addCell(node == null ? null : node.getHostAddress()); table.addCell(node == null ? "UNASSIGNED" : node.name()); @@ -185,6 +177,7 @@ public class RestAllocationAction extends AbstractCatAction { table.addCell(null); table.addCell(null); table.addCell(null); + table.addCell(null); table.addCell("UNASSIGNED"); table.endRow(); } From 349a8be4fd9be31b2e264a660ec38b24fc1795b7 Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Tue, 14 Jan 2014 22:31:42 +0100 Subject: [PATCH 18/34] Consistent REST API changes for GETting data * Made GET mappings consistent, supporting * /{index}/_mappings/{type} * /{index}/_mapping/{type} * /_mapping/{type} * Added "mappings" in the JSON response to align it with other responses * Made GET warmers consistent, support /{index}/_warmers/{type} and /_warmer, /_warner/{name} as well as wildcards and _all notation * Made GET aliases consistent, support /{index}/_aliases/{name} and /_alias, /_aliases/{name} as well as wildcards and _all notation * Made GET settings consistent, added /{index}/_setting/{name}, /_settings/{name} as well as supportings wildcards in settings name * Returning empty JSON instead of a 404, if a specific warmer/ setting/alias/type is missing * Added a ton of spec tests for all of the above * Added a couple of more integration tests for several features Relates #4071 --- docs/reference/indices/get-settings.asciidoc | 6 +- rest-api-spec/api/indices.get_alias.json | 4 +- rest-api-spec/api/indices.get_aliases.json | 6 +- rest-api-spec/api/indices.get_mapping.json | 2 +- rest-api-spec/api/indices.get_settings.json | 6 +- rest-api-spec/api/indices.get_warmer.json | 4 +- .../test/indices.delete_alias/10_basic.yaml | 3 +- .../all_path_options.yaml | 16 +- .../test/indices.get_alias/10_basic.yaml | 211 +++++++++++++++++ .../test/indices.get_aliases/10_basic.yaml | 214 +++++++++++++++++ .../test/indices.get_mapping/10_basic.yaml | 179 +++++++++++++-- .../indices.get_mapping/20_missing_type.yaml | 6 +- .../test/indices.get_mapping/40_aliases.yaml | 26 +++ .../test/indices.get_settings/10_basic.yaml | 185 +++++++++++++-- .../test/indices.get_settings/20_aliases.yaml | 26 +++ .../test/indices.get_warmer/10_basic.yaml | 215 ++++++++++++++++++ .../test/indices.put_mapping/10_basic.yaml | 16 +- .../indices.put_mapping/all_path_options.yaml | 66 +++--- .../test/indices.put_warmer/10_basic.yaml | 116 +++++++++- .../test/indices.put_warmer/20_aliases.yaml | 30 +++ .../settings/get/GetSettingsRequest.java | 21 +- .../get/GetSettingsRequestBuilder.java | 4 +- .../get/TransportGetSettingsAction.java | 6 +- .../cluster/metadata/MetaData.java | 6 +- .../org/elasticsearch/common/Strings.java | 11 + .../org/elasticsearch/rest/RestRequest.java | 9 + .../alias/get/RestGetAliasesAction.java | 9 +- .../get/RestGetIndicesAliasesAction.java | 26 ++- .../mapping/get/RestGetMappingAction.java | 17 +- .../settings/RestGetSettingsAction.java | 15 +- .../warmer/get/RestGetWarmerAction.java | 6 +- .../action/support/RestXContentBuilder.java | 4 + .../warmer/SimpleIndicesWarmerTests.java | 30 +++ 33 files changed, 1347 insertions(+), 154 deletions(-) create mode 100644 rest-api-spec/test/indices.get_alias/10_basic.yaml create mode 100644 rest-api-spec/test/indices.get_aliases/10_basic.yaml create mode 100644 rest-api-spec/test/indices.get_mapping/40_aliases.yaml create mode 100644 rest-api-spec/test/indices.get_settings/20_aliases.yaml create mode 100644 rest-api-spec/test/indices.get_warmer/10_basic.yaml create mode 100644 rest-api-spec/test/indices.put_warmer/20_aliases.yaml diff --git a/docs/reference/indices/get-settings.asciidoc b/docs/reference/indices/get-settings.asciidoc index 0a0e10df861..a5950c2ee4b 100644 --- a/docs/reference/indices/get-settings.asciidoc +++ b/docs/reference/indices/get-settings.asciidoc @@ -39,12 +39,12 @@ curl -XGET 'http://localhost:9200/my-index/_settings?prefix=index.' curl -XGET 'http://localhost:9200/_all/_settings?prefix=index.routing.allocation.' -curl -XGET 'http://localhost:9200/2013-*/_settings?prefix=index.merge.' +curl -XGET 'http://localhost:9200/2013-*/_settings?name=index.merge.*' -curl -XGET 'http://localhost:9200/2013-*/index.merge./_settings' +curl -XGET 'http://localhost:9200/2013-*/_settings/index.merge.*' -------------------------------------------------- The first example returns all index settings the start with `index.` in the index `my-index`, the second example gets all index settings that start with `index.routing.allocation.` for all indices, lastly the third example returns all index settings that start with `index.merge.` -in indices that start with `2013-`. \ No newline at end of file +in indices that start with `2013-`. diff --git a/rest-api-spec/api/indices.get_alias.json b/rest-api-spec/api/indices.get_alias.json index 61126fa963e..e7fe6e4a500 100644 --- a/rest-api-spec/api/indices.get_alias.json +++ b/rest-api-spec/api/indices.get_alias.json @@ -3,8 +3,8 @@ "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html", "methods": ["GET"], "url": { - "path": "/_alias/{name}", - "paths": ["/_alias/{name}", "/{index}/_alias/{name}", "/{index}/_alias"], + "path": "/_alias/", + "paths": [ "/_alias", "/_alias/{name}", "/{index}/_alias/{name}", "/{index}/_alias"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/api/indices.get_aliases.json b/rest-api-spec/api/indices.get_aliases.json index 508c94e3c0c..f3abd7a5602 100644 --- a/rest-api-spec/api/indices.get_aliases.json +++ b/rest-api-spec/api/indices.get_aliases.json @@ -4,11 +4,15 @@ "methods": ["GET"], "url": { "path": "/_aliases", - "paths": ["/_aliases", "/{index}/_aliases"], + "paths": ["/_aliases", "/{index}/_aliases", "/{index}/_aliases/{name}", "/_aliases/{name}" ], "parts": { "index": { "type" : "list", "description" : "A comma-separated list of index names to filter aliases" + }, + "name": { + "type" : "list", + "description" : "A comma-separated list of alias names to filter" } }, "params": { diff --git a/rest-api-spec/api/indices.get_mapping.json b/rest-api-spec/api/indices.get_mapping.json index 75cf384ff32..2438deec57a 100644 --- a/rest-api-spec/api/indices.get_mapping.json +++ b/rest-api-spec/api/indices.get_mapping.json @@ -4,7 +4,7 @@ "methods": ["GET"], "url": { "path": "/_mapping", - "paths": ["/_mapping", "/{index}/_mapping", "/{index}/{type}/_mapping"], + "paths": ["/_mapping", "/{index}/_mapping", "/_mapping/{type}", "/{index}/_mapping/{type}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/api/indices.get_settings.json b/rest-api-spec/api/indices.get_settings.json index ae6b6e33c38..b02434db13e 100644 --- a/rest-api-spec/api/indices.get_settings.json +++ b/rest-api-spec/api/indices.get_settings.json @@ -4,15 +4,15 @@ "methods": ["GET"], "url": { "path": "/_settings", - "paths": ["/_settings", "/{index}/_settings", "/{index}/{prefix}/_settings"], + "paths": ["/_settings", "/{index}/_settings", "/{index}/_settings/{name}", "/_settings/{name}"], "parts": { "index": { "type" : "list", "description" : "A comma-separated list of index names; use `_all` or empty string to perform the operation on all indices" }, - "prefix": { + "name": { "type" : "string", - "description" : "The prefix all settings must have in order to be included" + "description" : "The name of the settings that should be included" } }, "params": { diff --git a/rest-api-spec/api/indices.get_warmer.json b/rest-api-spec/api/indices.get_warmer.json index ae17b957ef6..40a1e834519 100644 --- a/rest-api-spec/api/indices.get_warmer.json +++ b/rest-api-spec/api/indices.get_warmer.json @@ -3,8 +3,8 @@ "documentation": "http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html", "methods": ["GET"], "url": { - "path": "/{index}/_warmer", - "paths": ["/{index}/_warmer", "/{index}/_warmer/{name}", "/{index}/{type}/_warmer/{name}"], + "path": "/_warmer", + "paths": [ "/_warmer", "/{index}/_warmer", "/{index}/_warmer/{name}", "/_warmer/{name}", "/{index}/{type}/_warmer/{name}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/test/indices.delete_alias/10_basic.yaml b/rest-api-spec/test/indices.delete_alias/10_basic.yaml index 9f3535f7ef6..87f61efc112 100644 --- a/rest-api-spec/test/indices.delete_alias/10_basic.yaml +++ b/rest-api-spec/test/indices.delete_alias/10_basic.yaml @@ -25,7 +25,8 @@ name: testali - do: - catch: missing indices.get_alias: index: testind name: testali + + - match: { '': {}} diff --git a/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml b/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml index 2953a096435..a0ed0fdd419 100644 --- a/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml +++ b/rest-api-spec/test/indices.delete_warmer/all_path_options.yaml @@ -49,12 +49,7 @@ setup: name: test_warmer1 - do: - catch: missing - indices.get_warmer: { index: _all, name: 'test_warmer1' } - - - do: - indices.get_warmer: { index: _all, name: 'test_warmer2' } - + indices.get_warmer: {} - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} @@ -68,12 +63,7 @@ setup: name: test_warmer1 - do: - catch: missing - indices.get_warmer: { index: _all, name: 'test_warmer1' } - - - do: - indices.get_warmer: { index: _all, name: 'test_warmer2' } - + indices.get_warmer: {} - match: {test_index1.warmers.test_warmer2.source.query.match_all: {}} - match: {test_index2.warmers.test_warmer2.source.query.match_all: {}} @@ -146,7 +136,7 @@ setup: --- "check delete with index list and _all warmers": - do: - indices.delete_warmer: + indices.delete_warmer: index: "test_index1,test_index2" name: _all diff --git a/rest-api-spec/test/indices.get_alias/10_basic.yaml b/rest-api-spec/test/indices.get_alias/10_basic.yaml new file mode 100644 index 00000000000..21ff08c4f80 --- /dev/null +++ b/rest-api-spec/test/indices.get_alias/10_basic.yaml @@ -0,0 +1,211 @@ +--- +setup: + + - do: + indices.create: + index: test_index + + - do: + indices.create: + index: test_index_2 + + - do: + indices.put_alias: + index: test_index + name: test_alias + + - do: + indices.put_alias: + index: test_index + name: test_blias + + - do: + indices.put_alias: + index: test_index_2 + name: test_alias + + - do: + indices.put_alias: + index: test_index_2 + name: test_blias + +--- +"Get all aliases via /_alias": + + - do: + indices.get_alias: {} + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_blias: {}} + + +--- +"Get all aliases via /{index}/_alias/": + + - do: + indices.get_alias: + index: test_index + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get specific alias via /{index}/_alias/{name}": + + - do: + indices.get_alias: + index: test_index + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_alias/_all": + + - do: + indices.get_alias: + index: test_index + name: _all + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_alias/*": + + - do: + indices.get_alias: + index: test_index + name: '*' + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_alias/prefix*": + + - do: + indices.get_alias: + index: test_index + name: 'test_a*' + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_alias/name,name": + + - do: + indices.get_alias: + index: test_index + name: 'test_alias,test_blias' + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /_alias/{name}": + + - do: + indices.get_alias: + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /_all/_alias/{name}": + + - do: + indices.get_alias: + index: _all + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /*/_alias/{name}": + + - do: + indices.get_alias: + index: '*' + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /pref*/_alias/{name}": + + - do: + indices.get_alias: + index: '*2' + name: test_alias + + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_alias + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /name,name/_alias/{name}": + + - do: + indices.get_alias: + index: test_index,test_index_2 + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + + +--- +"Non-existent alias on an existing index returns an empty body": + + - do: + indices.get_alias: + index: test_index + name: non-existent + + - match: { '': {}} + +--- +"Existent and non-existent alias returns just the existing": + + - do: + indices.get_alias: + index: test_index + name: test_alias,non-existent + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.non-existent + +--- +"Getting alias on an non-existent index should return 404": + + - do: + catch: missing + indices.get_alias: + index: non-existent + name: foo + + + diff --git a/rest-api-spec/test/indices.get_aliases/10_basic.yaml b/rest-api-spec/test/indices.get_aliases/10_basic.yaml new file mode 100644 index 00000000000..0e235f223d5 --- /dev/null +++ b/rest-api-spec/test/indices.get_aliases/10_basic.yaml @@ -0,0 +1,214 @@ +--- +setup: + + - do: + indices.create: + index: test_index + + - do: + indices.create: + index: test_index_2 + + - do: + indices.put_alias: + index: test_index + name: test_alias + + - do: + indices.put_alias: + index: test_index + name: test_blias + + - do: + indices.put_alias: + index: test_index_2 + name: test_alias + + - do: + indices.put_alias: + index: test_index_2 + name: test_blias + +--- +"Get all aliases via /_aliases": + + - do: + indices.get_aliases: {} + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_blias: {}} + + +--- +"Get all aliases via /{index}/_aliases/": + + - do: + indices.get_aliases: + index: test_index + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get specific alias via /{index}/_aliases/{name}": + + - do: + indices.get_aliases: + index: test_index + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_aliases/_all": + + - do: + indices.get_aliases: + index: test_index + name: _all + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_aliases/*": + + - do: + indices.get_aliases: + index: test_index + name: '*' + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_aliases/prefix*": + + - do: + indices.get_aliases: + index: test_index + name: 'test_a*' + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2 + +--- +"Get aliases via /{index}/_aliases/name,name": + + - do: + indices.get_aliases: + index: test_index + name: 'test_alias,test_blias' + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index.aliases.test_blias: {}} + - is_false: test_index_2 + +--- +"Get aliases via /_aliases/{name}": + + - do: + indices.get_aliases: + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /_all/_aliases/{name}": + + - do: + indices.get_aliases: + index: _all + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /*/_aliases/{name}": + + - do: + indices.get_aliases: + index: '*' + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /pref*/_aliases/{name}": + + - do: + indices.get_aliases: + index: '*2' + name: test_alias + + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_alias + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + +--- +"Get aliases via /name,name/_aliases/{name}": + + - do: + indices.get_aliases: + index: test_index,test_index_2 + name: test_alias + + - match: {test_index.aliases.test_alias: {}} + - match: {test_index_2.aliases.test_alias: {}} + - is_false: test_index.aliases.test_blias + - is_false: test_index_2.aliases.test_blias + + +--- +"Non-existent alias on an existing index returns matching indcies": + + - do: + indices.get_aliases: + index: test_index + name: non-existent + + - match: { test_index.aliases: {}} + +--- +"Existent and non-existent alias returns just the existing": + + - do: + indices.get_aliases: + index: test_index + name: test_alias,non-existent + + - match: {test_index.aliases.test_alias: {}} + - is_false: test_index.aliases.non-existent + +--- +"Getting alias on an non-existent index should return 404": + + - skip: + version: 1 - 999 + reason: not implemented yet + - do: + catch: missing + indices.get_aliases: + index: non-existent + name: foo + + + diff --git a/rest-api-spec/test/indices.get_mapping/10_basic.yaml b/rest-api-spec/test/indices.get_mapping/10_basic.yaml index 40416435bb9..4ed9b761bb7 100644 --- a/rest-api-spec/test/indices.get_mapping/10_basic.yaml +++ b/rest-api-spec/test/indices.get_mapping/10_basic.yaml @@ -2,31 +2,172 @@ setup: - do: indices.create: - index: test_index + index: test_1 body: mappings: - test_type: - properties: - text: - type: string - analyzer: whitespace + type_1: {} + type_2: {} + - do: + indices.create: + index: test_2 + body: + mappings: + type_2: {} + type_3: {} --- -"Get index mapping": - - do: - indices.get_mapping: - index: test_index +"Get /_mapping": - - match: {test_index.test_type.properties.text.type: string} - - match: {test_index.test_type.properties.text.analyzer: whitespace} + - do: + indices.get_mapping: {} + + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_3.properties: {}} --- -"Get type mapping": +"Get /{index}/_mapping": - - do: - indices.get_mapping: - index: test_index - type: test_type + - do: + indices.get_mapping: + index: test_1 - - match: {test_index.test_type.properties.text.type: string} - - match: {test_index.test_type.properties.text.analyzer: whitespace} + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_2 + + +--- +"Get /{index}/_mapping/_all": + + - do: + indices.get_mapping: + index: test_1 + type: _all + + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_2 + +--- +"Get /{index}/_mapping/*": + + - do: + indices.get_mapping: + index: test_1 + type: '*' + + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_2 + +--- +"Get /{index}/_mapping/{type}": + + - do: + indices.get_mapping: + index: test_1 + type: type_1 + + - match: { test_1.mappings.type_1.properties: {}} + - is_false: test_1.mappings.type_2 + - is_false: test_2 + +--- +"Get /{index}/_mapping/{type,type}": + + - do: + indices.get_mapping: + index: test_1 + type: type_1,type_2 + + - match: { test_1.mappings.type_1.properties: {}} + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_2 + +--- +"Get /{index}/_mapping/{type*}": + + - do: + indices.get_mapping: + index: test_1 + type: '*2' + + - match: { test_1.mappings.type_2.properties: {}} + - is_false: test_1.mappings.type_1 + - is_false: test_2 + +--- +"Get /_mapping/{type}": + + - do: + indices.get_mapping: + type: type_2 + + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1.mappings.type_1 + - is_false: test_2.mappings.type_3 + +--- +"Get /_all/_mapping/{type}": + + - do: + indices.get_mapping: + index: _all + type: type_2 + + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1.mappings.type_1 + - is_false: test_2.mappings.type_3 + +--- +"Get /*/_mapping/{type}": + + - do: + indices.get_mapping: + index: '*' + type: type_2 + + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1.mappings.type_1 + - is_false: test_2.mappings.type_3 + +--- +"Get /{index}/_mapping/{type}": + + - do: + indices.get_mapping: + index: test_2 + type: type_2 + + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1 + - is_false: test_2.mappings.type_3 + +--- +"Get /index,index/_mapping/{type}": + + - do: + indices.get_mapping: + index: test_1,test_2 + type: type_2 + + - match: { test_1.mappings.type_2.properties: {}} + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_2.mappings.type_3 + +--- +"Get /index*/_mapping/{type}": + + - do: + indices.get_mapping: + index: '*2' + type: type_2 + + - match: { test_2.mappings.type_2.properties: {}} + - is_false: test_1 + - is_false: test_2.mappings.type_3 diff --git a/rest-api-spec/test/indices.get_mapping/20_missing_type.yaml b/rest-api-spec/test/indices.get_mapping/20_missing_type.yaml index e7d748acf92..91b1883197b 100644 --- a/rest-api-spec/test/indices.get_mapping/20_missing_type.yaml +++ b/rest-api-spec/test/indices.get_mapping/20_missing_type.yaml @@ -1,5 +1,5 @@ --- -"Raise 404 when type doesn't exist": +"Return empty response when type doesn't exist": - do: indices.create: index: test_index @@ -12,8 +12,8 @@ analyzer: whitespace - do: - catch: missing indices.get_mapping: index: test_index type: not_test_type - + + - match: { '': {}} diff --git a/rest-api-spec/test/indices.get_mapping/40_aliases.yaml b/rest-api-spec/test/indices.get_mapping/40_aliases.yaml new file mode 100644 index 00000000000..9dff6edc9dc --- /dev/null +++ b/rest-api-spec/test/indices.get_mapping/40_aliases.yaml @@ -0,0 +1,26 @@ +--- +"Getting mapping for aliases should return the real index as key": + + - do: + indices.create: + index: test_index + body: + mappings: + test_type: + properties: + text: + type: string + analyzer: whitespace + + - do: + indices.put_alias: + index: test_index + name: test_alias + + - do: + indices.get_mapping: + index: test_alias + + - match: {test_index.mappings.test_type.properties.text.type: string} + - match: {test_index.mappings.test_type.properties.text.analyzer: whitespace} + diff --git a/rest-api-spec/test/indices.get_settings/10_basic.yaml b/rest-api-spec/test/indices.get_settings/10_basic.yaml index 50ff2cfc041..5e9ce897298 100644 --- a/rest-api-spec/test/indices.get_settings/10_basic.yaml +++ b/rest-api-spec/test/indices.get_settings/10_basic.yaml @@ -1,32 +1,167 @@ --- -"Test get indices settings": +setup: - do: - indices.create: - index: test-index - body: - settings: - index: - refresh_interval: -1 - number_of_shards: 2 - number_of_replicas: 3 - + indices.create: + index: test_1 - do: - indices.get_settings: - index: test-index + indices.create: + index: test_2 - - match: - test-index.settings.index.number_of_replicas: "3" - - match: - test-index.settings.index.number_of_shards: "2" - - match: - test-index.settings.index.refresh_interval: "-1" +--- +"Get /_settings": - - do: - indices.get_settings: + - do: + indices.get_settings: {} + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - match: { test_2.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_replicas: "1"} + +--- +"Get /{index}/_settings": + + - do: + indices.get_settings: + index: test_1 + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - is_false: test_2 + + +--- +"Get /{index}/_settings/_all": + + - do: + indices.get_settings: + index: test_1 + name: _all + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - is_false: test_2 + +--- +"Get /{index}/_settings/*": + + - do: + indices.get_settings: + index: test_1 + name: '*' + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - is_false: test_2 + +--- +"Get /{index}/_settings/{name}": + + - do: + indices.get_settings: + index: test_1 + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2 + +--- +"Get /{index}/_settings/{name,name}": + + - do: + indices.get_settings: + index: test_1 + name: index.number_of_shards,index.number_of_replicas + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_1.settings.index.number_of_replicas: "1"} + - is_false: test_2 + +--- +"Get /{index}/_settings/{name*}": + + - do: + indices.get_settings: + index: test_1 + name: 'index.number_of_s*' + + - match: { test_1.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2 + +--- +"Get /_settings/{name}": + + - do: + indices.get_settings: + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2.settings.index.number_of_replicas + +--- +"Get /_all/_settings/{name}": + + - do: + indices.get_settings: index: _all - prefix: index.number + name: index.number_of_shards - - match: - test-index.settings.index.number_of_replicas: "3" - - match: - test-index.settings.index.number_of_shards: "2" + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2.settings.index.number_of_replicas + + +--- +"Get /*/_settings/{name}": + + - do: + indices.get_settings: + index: '*' + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2.settings.index.number_of_replicas + +--- +"Get /{index}/_settings/{name}": + + - do: + indices.get_settings: + index: test_1 + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2 + +--- +"Get /index,index/_settings/{name}": + + - do: + indices.get_settings: + index: test_1,test_2 + name: index.number_of_shards + + - match: { test_1.settings.index.number_of_shards: "5"} + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1.settings.index.number_of_replicas + - is_false: test_2.settings.index.number_of_replicas + +--- +"Get /index*/_settings/{name}": + + - do: + indices.get_settings: + index: '*2' + name: index.number_of_shards + + - match: { test_2.settings.index.number_of_shards: "5"} + - is_false: test_1 + - is_false: test_2.settings.index.number_of_replicas diff --git a/rest-api-spec/test/indices.get_settings/20_aliases.yaml b/rest-api-spec/test/indices.get_settings/20_aliases.yaml new file mode 100644 index 00000000000..da7678202ed --- /dev/null +++ b/rest-api-spec/test/indices.get_settings/20_aliases.yaml @@ -0,0 +1,26 @@ +--- +"Getting settings for aliases should return the real index as key": + + - do: + indices.create: + index: test-index + body: + settings: + index: + refresh_interval: -1 + number_of_shards: 2 + number_of_replicas: 3 + + - do: + indices.put_alias: + index: test-index + name: test-alias + + - do: + indices.get_settings: + index: test-alias + + - match: { test-index.settings.index.number_of_replicas: "3" } + - match: { test-index.settings.index.number_of_shards: "2" } + - match: { test-index.settings.index.refresh_interval: "-1" } + diff --git a/rest-api-spec/test/indices.get_warmer/10_basic.yaml b/rest-api-spec/test/indices.get_warmer/10_basic.yaml new file mode 100644 index 00000000000..ed13f90c21b --- /dev/null +++ b/rest-api-spec/test/indices.get_warmer/10_basic.yaml @@ -0,0 +1,215 @@ +--- +setup: + - do: + indices.create: + index: test_1 + - do: + indices.create: + index: test_2 + + - do: + cluster.health: + wait_for_status: yellow + - do: + indices.put_warmer: + index: test_1 + name: warmer_1 + body: { query: { match_all: { }}} + + - do: + indices.put_warmer: + index: test_1 + name: warmer_2 + body: { query: { match_all: { }}} + + - do: + indices.put_warmer: + index: test_2 + name: warmer_2 + body: { query: { match_all: { }}} + - do: + indices.put_warmer: + index: test_2 + name: warmer_3 + body: { query: { match_all: { }}} + + - do: + indices.refresh: {} + +--- +"Get /_warmer": + + - do: + indices.get_warmer: {} + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_3.source.query.match_all: {}} + +--- +"Get /{index}/_warmer": + + - do: + indices.get_warmer: + index: test_1 + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2 + + +--- +"Get /{index}/_warmer/_all": + + - do: + indices.get_warmer: + index: test_1 + name: _all + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2 + +--- +"Get /{index}/_warmer/*": + + - do: + indices.get_warmer: + index: test_1 + name: '*' + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2 + +--- +"Get /{index}/_warmer/{name}": + + - do: + indices.get_warmer: + index: test_1 + name: warmer_1 + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_2 + - is_false: test_2 + +--- +"Get /{index}/_warmer/{name,name}": + + - do: + indices.get_warmer: + index: test_1 + name: warmer_1,warmer_2 + + - match: { test_1.warmers.warmer_1.source.query.match_all: {}} + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2 + +--- +"Get /{index}/_warmer/{name*}": + + - do: + indices.get_warmer: + index: test_1 + name: '*2' + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_1 + - is_false: test_2 + +--- +"Get /_warmer/{name}": + + - do: + indices.get_warmer: + name: warmer_2 + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Get /_all/_warmer/{name}": + + - do: + indices.get_warmer: + index: _all + name: warmer_2 + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Get /*/_warmer/{name}": + + - do: + indices.get_warmer: + index: '*' + name: warmer_2 + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1.warmers.warmer_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Get /{index}/_warmer/{name}": + + - do: + indices.get_warmer: + index: test_2 + name: warmer_2 + + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Get /index,index/_warmer/{name}": + + - do: + indices.get_warmer: + index: test_1,test_2 + name: warmer_2 + + - match: { test_1.warmers.warmer_2.source.query.match_all: {}} + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_2.warmers.warmer_3 + +--- +"Get /index*/_warmer/{name}": + + - do: + indices.get_warmer: + index: '*2' + name: warmer_2 + + - match: { test_2.warmers.warmer_2.source.query.match_all: {}} + - is_false: test_1 + - is_false: test_2.warmers.warmer_3 + +--- +"Empty response when no matching warmer": + + - do: + indices.get_warmer: + index: '*' + name: non_existent + + - match: { '': {}} + +--- +"Throw 404 on missing index": + + - do: + catch: missing + indices.get_warmer: + index: non_existent + name: '*' + + diff --git a/rest-api-spec/test/indices.put_mapping/10_basic.yaml b/rest-api-spec/test/indices.put_mapping/10_basic.yaml index c227da8673b..e02b948f936 100644 --- a/rest-api-spec/test/indices.put_mapping/10_basic.yaml +++ b/rest-api-spec/test/indices.put_mapping/10_basic.yaml @@ -22,10 +22,10 @@ indices.get_mapping: index: test_index - - match: {test_index.test_type.properties.text1.type: string} - - match: {test_index.test_type.properties.text1.analyzer: whitespace} - - match: {test_index.test_type.properties.text2.type: string} - - match: {test_index.test_type.properties.text2.analyzer: whitespace} + - match: {test_index.mappings.test_type.properties.text1.type: string} + - match: {test_index.mappings.test_type.properties.text1.analyzer: whitespace} + - match: {test_index.mappings.test_type.properties.text2.type: string} + - match: {test_index.mappings.test_type.properties.text2.analyzer: whitespace} - do: indices.put_mapping: @@ -56,7 +56,7 @@ indices.get_mapping: index: test_index - - match: {test_index.test_type.properties.text1.type: string} - - match: {test_index.test_type.properties.text1.fields.text_raw.index: not_analyzed} - - match: {test_index.test_type.properties.text2.type: string} - - match: {test_index.test_type.properties.text2.fields.text_raw.index: not_analyzed} + - match: {test_index.mappings.test_type.properties.text1.type: string} + - match: {test_index.mappings.test_type.properties.text1.fields.text_raw.index: not_analyzed} + - match: {test_index.mappings.test_type.properties.text2.type: string} + - match: {test_index.mappings.test_type.properties.text2.fields.text_raw.index: not_analyzed} diff --git a/rest-api-spec/test/indices.put_mapping/all_path_options.yaml b/rest-api-spec/test/indices.put_mapping/all_path_options.yaml index db0df345c5d..1c26184513d 100644 --- a/rest-api-spec/test/indices.put_mapping/all_path_options.yaml +++ b/rest-api-spec/test/indices.put_mapping/all_path_options.yaml @@ -37,13 +37,13 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo: {}} + - is_false: foo --- "put mapping in _all index": @@ -62,14 +62,14 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo.test_type.properties.text.type: string} - - match: {foo.test_type.properties.text.analyzer: whitespace} + - match: {foo.mappings.test_type.properties.text.type: string} + - match: {foo.mappings.test_type.properties.text.analyzer: whitespace} --- "put mapping in * index": @@ -87,14 +87,14 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo.test_type.properties.text.type: string} - - match: {foo.test_type.properties.text.analyzer: whitespace} + - match: {foo.mappings.test_type.properties.text.type: string} + - match: {foo.mappings.test_type.properties.text.analyzer: whitespace} --- "put mapping in prefix* index": @@ -112,13 +112,13 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo: {}} + - is_false: foo --- "put mapping in list of indices": @@ -136,13 +136,13 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo: {}} + - is_false: foo --- "put mapping with blank index": @@ -159,14 +159,14 @@ setup: - do: indices.get_mapping: {} - - match: {test_index1.test_type.properties.text.type: string} - - match: {test_index1.test_type.properties.text.analyzer: whitespace} + - match: {test_index1.mappings.test_type.properties.text.type: string} + - match: {test_index1.mappings.test_type.properties.text.analyzer: whitespace} - - match: {test_index2.test_type.properties.text.type: string} - - match: {test_index2.test_type.properties.text.analyzer: whitespace} + - match: {test_index2.mappings.test_type.properties.text.type: string} + - match: {test_index2.mappings.test_type.properties.text.analyzer: whitespace} - - match: {foo.test_type.properties.text.type: string} - - match: {foo.test_type.properties.text.analyzer: whitespace} + - match: {foo.mappings.test_type.properties.text.type: string} + - match: {foo.mappings.test_type.properties.text.analyzer: whitespace} --- "put mapping with mising type": diff --git a/rest-api-spec/test/indices.put_warmer/10_basic.yaml b/rest-api-spec/test/indices.put_warmer/10_basic.yaml index 0e2a6223605..44313aa9bfa 100644 --- a/rest-api-spec/test/indices.put_warmer/10_basic.yaml +++ b/rest-api-spec/test/indices.put_warmer/10_basic.yaml @@ -1,18 +1,24 @@ --- -"Basic test for warmers": +setup: - do: indices.create: index: test_index + - do: + indices.create: + index: test_idx + - do: cluster.health: wait_for_status: yellow - do: - catch: missing - indices.get_warmer: - index: test_index - name: test_warmer + indices.put_warmer: + index: test_idx + name: test_warmer2 + body: + query: + match_all: {} - do: indices.put_warmer: @@ -22,6 +28,8 @@ query: match_all: {} +--- +"Basic test for warmers": - do: indices.get_warmer: index: test_index @@ -35,7 +43,103 @@ name: test_warmer - do: - catch: missing indices.get_warmer: index: test_index name: test_warmer + + - match: { '': {}} + +--- +"Getting all warmers via /_warmer should work": + + - do: + indices.get_warmer: {} + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + + +--- +"Getting warmers for several indices should work using *": + + - do: + indices.get_warmer: + index: '*' + name: '*' + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting warmers for several indices should work using _all": + + - do: + indices.get_warmer: + index: _all + name: _all + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting all warmers without specifying index should work": + + - do: + indices.get_warmer: + name: _all + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting warmers for several indices should work using prefix*": + + - do: + indices.get_warmer: + index: test_i* + name: test_w* + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting warmers for several indices should work using comma-separated lists": + + - do: + indices.get_warmer: + index: test_index,test_idx + name: test_warmer,test_warmer2 + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - match: {test_idx.warmers.test_warmer2.source.query.match_all: {}} + +--- +"Getting a non-existent warmer on an existing index should return an empty body": + + - do: + indices.get_warmer: + index: test_index + name: non-existent + + - match: { '': {}} + +--- +"Getting an existent and non-existent warmer should return the existent and no data about the non-existent warmer": + + - do: + indices.get_warmer: + index: test_index + name: test_warmer,non-existent + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + - is_false: test_index.warmers.non-existent + +--- +"Getting warmer on an non-existent index should return 404": + + - do: + catch: missing + indices.get_warmer: + index: non-existent + name: foo + diff --git a/rest-api-spec/test/indices.put_warmer/20_aliases.yaml b/rest-api-spec/test/indices.put_warmer/20_aliases.yaml new file mode 100644 index 00000000000..96d734475ac --- /dev/null +++ b/rest-api-spec/test/indices.put_warmer/20_aliases.yaml @@ -0,0 +1,30 @@ +--- +"Getting warmer for aliases should return the real index as key": + + - do: + indices.create: + index: test_index + + - do: + cluster.health: + wait_for_status: yellow + + - do: + indices.put_warmer: + index: test_index + name: test_warmer + body: + query: + match_all: {} + + - do: + indices.put_alias: + index: test_index + name: test_alias + + - do: + indices.get_warmer: + index: test_alias + + - match: {test_index.warmers.test_warmer.source.query.match_all: {}} + diff --git a/src/main/java/org/elasticsearch/action/admin/indices/settings/get/GetSettingsRequest.java b/src/main/java/org/elasticsearch/action/admin/indices/settings/get/GetSettingsRequest.java index 3025caee581..311a60d262c 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/settings/get/GetSettingsRequest.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/settings/get/GetSettingsRequest.java @@ -20,6 +20,7 @@ package org.elasticsearch.action.admin.indices.settings.get; import org.elasticsearch.action.ActionRequestValidationException; +import org.elasticsearch.action.ValidateActions; import org.elasticsearch.action.support.IndicesOptions; import org.elasticsearch.action.support.master.MasterNodeOperationRequest; import org.elasticsearch.common.Strings; @@ -34,7 +35,7 @@ public class GetSettingsRequest extends MasterNodeOperationRequest entry : settings.getAsMap().entrySet()) { - if (entry.getKey().startsWith(request.prefix())) { + if (Regex.simpleMatch(request.names(), entry.getKey())) { settingsBuilder.put(entry.getKey(), entry.getValue()); } } diff --git a/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java b/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java index b2310d2532c..d20f0b2775c 100644 --- a/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java +++ b/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java @@ -376,12 +376,14 @@ public class MetaData implements Iterable { return indexMapBuilder.build(); } - public ImmutableOpenMap> findWarmers(String[] concreteIndices, final String[] types, final String[] warmers) { - assert warmers != null; + public ImmutableOpenMap> findWarmers(String[] concreteIndices, final String[] types, final String[] uncheckedWarmers) { + assert uncheckedWarmers != null; assert concreteIndices != null; if (concreteIndices.length == 0) { return ImmutableOpenMap.of(); } + // special _all check to behave the same like not specifying anything for the warmers (not for the indices) + final String[] warmers = Strings.isAllOrWildcard(uncheckedWarmers) ? Strings.EMPTY_ARRAY : uncheckedWarmers; ImmutableOpenMap.Builder> mapBuilder = ImmutableOpenMap.builder(); Iterable intersection = HppcMaps.intersection(ObjectOpenHashSet.from(concreteIndices), indices.keys()); diff --git a/src/main/java/org/elasticsearch/common/Strings.java b/src/main/java/org/elasticsearch/common/Strings.java index aa85afa294d..45aa62f4246 100644 --- a/src/main/java/org/elasticsearch/common/Strings.java +++ b/src/main/java/org/elasticsearch/common/Strings.java @@ -26,6 +26,7 @@ import org.apache.lucene.util.BytesRef; import org.apache.lucene.util.UnicodeUtil; import org.elasticsearch.ElasticsearchIllegalStateException; import org.elasticsearch.common.io.FastStringReader; +import org.elasticsearch.common.util.CollectionUtils; import java.io.BufferedReader; import java.io.IOException; @@ -1565,4 +1566,14 @@ public class Strings { return s.substring(beginIndex, endIndex); } } + + /** + * If an array only consists of zero or one element, which is "*" or "_all" return an empty array + * which is usually used as everything + */ + public static boolean isAllOrWildcard(String[] data) { + return CollectionUtils.isEmpty(data) || + data.length == 1 && ("_all".equals(data[0]) || "*".equals(data[0])); + } + } diff --git a/src/main/java/org/elasticsearch/rest/RestRequest.java b/src/main/java/org/elasticsearch/rest/RestRequest.java index 781999bb31b..efd411e3742 100644 --- a/src/main/java/org/elasticsearch/rest/RestRequest.java +++ b/src/main/java/org/elasticsearch/rest/RestRequest.java @@ -158,4 +158,13 @@ public abstract class RestRequest implements ToXContent.Params { } return Strings.splitStringByCommaToArray(value); } + + public String[] paramAsStringArrayOrEmptyIfAll(String key) { + String[] params = paramAsStringArray(key, Strings.EMPTY_ARRAY); + if (Strings.isAllOrWildcard(params)) { + return Strings.EMPTY_ARRAY; + } + return params; + } + } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java index 5d7f71635bb..6357395ef41 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetAliasesAction.java @@ -49,6 +49,7 @@ public class RestGetAliasesAction extends BaseRestHandler { @Inject public RestGetAliasesAction(Settings settings, Client client, RestController controller) { super(settings, client); + controller.registerHandler(GET, "/_alias/", this); controller.registerHandler(GET, "/_alias/{name}", this); controller.registerHandler(GET, "/{index}/_alias/{name}", this); controller.registerHandler(GET, "/{index}/_alias", this); @@ -56,7 +57,7 @@ public class RestGetAliasesAction extends BaseRestHandler { @Override public void handleRequest(final RestRequest request, final RestChannel channel) { - String[] aliases = request.paramAsStringArray("name", Strings.EMPTY_ARRAY); + final String[] aliases = request.paramAsStringArrayOrEmptyIfAll("name"); final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); final GetAliasesRequest getAliasesRequest = new GetAliasesRequest(aliases); getAliasesRequest.indices(indices); @@ -68,7 +69,11 @@ public class RestGetAliasesAction extends BaseRestHandler { public void onResponse(GetAliasesResponse response) { try { XContentBuilder builder = RestXContentBuilder.restContentBuilder(request); - if (response.getAliases().isEmpty()) { + // empty body, if indices were specified but no aliases were + if (indices.length > 0 && response.getAliases().isEmpty()) { + channel.sendResponse(new XContentRestResponse(request, OK, RestXContentBuilder.emptyBuilder(request))); + return; + } else if (response.getAliases().isEmpty()) { String message = String.format(Locale.ROOT, "alias [%s] missing", toNamesString(getAliasesRequest.aliases())); builder.startObject() .field("error", message) diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java index c5122834a93..91c17c8dad8 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/alias/get/RestGetIndicesAliasesAction.java @@ -29,6 +29,7 @@ import org.elasticsearch.cluster.metadata.IndexMetaData; import org.elasticsearch.cluster.metadata.MetaData; import org.elasticsearch.common.Strings; import org.elasticsearch.common.inject.Inject; +import org.elasticsearch.common.regex.Regex; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.xcontent.ToXContent; import org.elasticsearch.common.xcontent.XContentBuilder; @@ -37,6 +38,7 @@ import org.elasticsearch.rest.action.support.RestXContentBuilder; import java.io.IOException; +import static org.elasticsearch.common.Strings.isAllOrWildcard; import static org.elasticsearch.rest.RestRequest.Method.GET; import static org.elasticsearch.rest.RestStatus.OK; @@ -50,16 +52,19 @@ public class RestGetIndicesAliasesAction extends BaseRestHandler { super(settings, client); controller.registerHandler(GET, "/_aliases", this); controller.registerHandler(GET, "/{index}/_aliases", this); + controller.registerHandler(GET, "/{index}/_aliases/{name}", this); + controller.registerHandler(GET, "/_aliases/{name}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); + final String[] aliases = Strings.splitStringByCommaToArray(request.param("name")); ClusterStateRequest clusterStateRequest = Requests.clusterStateRequest() - .routingTable(false) - .nodes(false) - .indices(indices); + .routingTable(false) + .nodes(false) + .indices(indices); clusterStateRequest.listenerThreaded(false); @@ -71,20 +76,22 @@ public class RestGetIndicesAliasesAction extends BaseRestHandler { XContentBuilder builder = RestXContentBuilder.restContentBuilder(request); builder.startObject(); + final boolean isAllAliasesRequested = isAllOrWildcard(aliases); for (IndexMetaData indexMetaData : metaData) { builder.startObject(indexMetaData.index(), XContentBuilder.FieldCaseConversion.NONE); - builder.startObject("aliases"); - for (ObjectCursor cursor : indexMetaData.aliases().values()) { - AliasMetaData.Builder.toXContent(cursor.value, builder, ToXContent.EMPTY_PARAMS); - } - builder.endObject(); + for (ObjectCursor cursor : indexMetaData.aliases().values()) { + if (isAllAliasesRequested || Regex.simpleMatch(aliases, cursor.value.alias())) { + AliasMetaData.Builder.toXContent(cursor.value, builder, ToXContent.EMPTY_PARAMS); + } + } + + builder.endObject(); builder.endObject(); } builder.endObject(); - channel.sendResponse(new XContentRestResponse(request, OK, builder)); } catch (Throwable e) { onFailure(e); @@ -101,4 +108,5 @@ public class RestGetIndicesAliasesAction extends BaseRestHandler { } }); } + } \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java index 57909f5bfde..a5fef39bb0d 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetMappingAction.java @@ -31,6 +31,7 @@ import org.elasticsearch.common.collect.ImmutableOpenMap; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.common.xcontent.XContentBuilderString; import org.elasticsearch.index.Index; import org.elasticsearch.indices.IndexMissingException; import org.elasticsearch.indices.TypeMissingException; @@ -53,12 +54,15 @@ public class RestGetMappingAction extends BaseRestHandler { controller.registerHandler(GET, "/_mapping", this); controller.registerHandler(GET, "/{index}/_mapping", this); controller.registerHandler(GET, "/{index}/{type}/_mapping", this); + controller.registerHandler(GET, "/{index}/_mappings/{type}", this); + controller.registerHandler(GET, "/{index}/_mapping/{type}", this); + controller.registerHandler(GET, "/_mapping/{type}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); - final String[] types = Strings.splitStringByCommaToArray(request.param("type")); + final String[] types = request.paramAsStringArrayOrEmptyIfAll("type"); boolean local = request.paramAsBooleanOptional("local", false); GetMappingsRequest getMappingsRequest = new GetMappingsRequest(); getMappingsRequest.indices(indices).types(types).local(local); @@ -74,7 +78,7 @@ public class RestGetMappingAction extends BaseRestHandler { ImmutableOpenMap> mappingsByIndex = response.getMappings(); if (mappingsByIndex.isEmpty()) { if (indices.length != 0 && types.length != 0) { - channel.sendResponse(new XContentThrowableRestResponse(request, new TypeMissingException(new Index(indices[0]), types[0]))); + channel.sendResponse(new XContentRestResponse(request, OK, RestXContentBuilder.emptyBuilder(request))); } else if (indices.length != 0) { channel.sendResponse(new XContentThrowableRestResponse(request, new IndexMissingException(new Index(indices[0])))); } else if (types.length != 0) { @@ -87,12 +91,17 @@ public class RestGetMappingAction extends BaseRestHandler { } for (ObjectObjectCursor> indexEntry : mappingsByIndex) { + if (indexEntry.value.isEmpty()) { + continue; + } builder.startObject(indexEntry.key, XContentBuilder.FieldCaseConversion.NONE); + builder.startObject(Fields.MAPPINGS); for (ObjectObjectCursor typeEntry : indexEntry.value) { builder.field(typeEntry.key); builder.map(typeEntry.value.sourceAsMap()); } builder.endObject(); + builder.endObject(); } builder.endObject(); @@ -112,4 +121,8 @@ public class RestGetMappingAction extends BaseRestHandler { } }); } + + static class Fields { + static final XContentBuilderString MAPPINGS = new XContentBuilderString("mappings"); + } } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java index 506f2ffa9ac..d87aa821481 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/settings/RestGetSettingsAction.java @@ -46,34 +46,39 @@ public class RestGetSettingsAction extends BaseRestHandler { super(settings, client); controller.registerHandler(GET, "/_settings", this); controller.registerHandler(GET, "/{index}/_settings", this); - controller.registerHandler(GET, "/{index}/{prefix}/_settings", this); + controller.registerHandler(GET, "/{index}/_settings/{name}", this); + controller.registerHandler(GET, "/_settings/{name}", this); + controller.registerHandler(GET, "/{index}/_setting/{name}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { + final String[] names = request.paramAsStringArrayOrEmptyIfAll("name"); GetSettingsRequest getSettingsRequest = new GetSettingsRequest() .indices(Strings.splitStringByCommaToArray(request.param("index"))) .indicesOptions(IndicesOptions.fromRequest(request, IndicesOptions.strict())) - .prefix(request.param("prefix")); + .names(names); client.admin().indices().getSettings(getSettingsRequest, new ActionListener() { @Override public void onResponse(GetSettingsResponse getSettingsResponse) { try { - boolean foundAny = false; XContentBuilder builder = RestXContentBuilder.restContentBuilder(request); builder.startObject(); for (ObjectObjectCursor cursor : getSettingsResponse.getIndexToSettings()) { + // no settings, jump over it to shorten the response data + if (cursor.value.getAsMap().isEmpty()) { + continue; + } builder.startObject(cursor.key, XContentBuilder.FieldCaseConversion.NONE); - foundAny = true; builder.startObject(Fields.SETTINGS); cursor.value.toXContent(builder, request); builder.endObject(); builder.endObject(); } builder.endObject(); - channel.sendResponse(new XContentRestResponse(request, foundAny ? OK : NOT_FOUND, builder)); + channel.sendResponse(new XContentRestResponse(request, OK, builder)); } catch (IOException e) { onFailure(e); } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java index ae2ea99a541..0dbe4c528dd 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/warmer/get/RestGetWarmerAction.java @@ -48,9 +48,11 @@ public class RestGetWarmerAction extends BaseRestHandler { @Inject public RestGetWarmerAction(Settings settings, Client client, RestController controller) { super(settings, client); - + controller.registerHandler(GET, "/_warmer", this); + controller.registerHandler(GET, "/_warmer/{name}", this); controller.registerHandler(GET, "/{index}/_warmer", this); controller.registerHandler(GET, "/{index}/_warmer/{name}", this); + controller.registerHandler(GET, "/{index}/_warmers/{name}", this); controller.registerHandler(GET, "/{index}/{type}/_warmer/{name}", this); } @@ -70,7 +72,7 @@ public class RestGetWarmerAction extends BaseRestHandler { public void onResponse(GetWarmersResponse response) { try { if (indices.length > 0 && response.warmers().isEmpty()) { - channel.sendResponse(new XContentThrowableRestResponse(request, new IndexMissingException(new Index(indices[0])))); + channel.sendResponse(new XContentRestResponse(request, OK, RestXContentBuilder.emptyBuilder(request))); return; } diff --git a/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java b/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java index fbde560fcf3..cff25557a66 100644 --- a/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java +++ b/src/main/java/org/elasticsearch/rest/action/support/RestXContentBuilder.java @@ -67,6 +67,10 @@ public class RestXContentBuilder { return builder; } + public static XContentBuilder emptyBuilder(RestRequest request) throws IOException { + return restContentBuilder(request, request.hasContent() ? request.content() : null).startObject().endObject(); + } + /** * Directly writes the source to the output builder */ diff --git a/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java b/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java index e1bcac1d2a9..e24985b6ad9 100644 --- a/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java +++ b/src/test/java/org/elasticsearch/indices/warmer/SimpleIndicesWarmerTests.java @@ -231,6 +231,36 @@ public class SimpleIndicesWarmerTests extends ElasticsearchIntegrationTest { assertThat(getWarmerRuns(), equalTo(warmerRunsAfterDisabling)); } + @Test + public void gettingAllWarmersUsingAllAndWildcardsShouldWork() throws Exception { + client().admin().indices().prepareCreate("test") + .setSettings(ImmutableSettings.settingsBuilder().put("index.number_of_shards", 1, "index.number_of_replicas", 0)) + .execute().actionGet(); + ensureGreen(); + + PutWarmerResponse putWarmerResponse = client().admin().indices().preparePutWarmer("custom_warmer") + .setSearchRequest(client().prepareSearch("test").setTypes("test").setQuery(QueryBuilders.matchAllQuery())) + .execute().actionGet(); + assertThat(putWarmerResponse.isAcknowledged(), equalTo(true)); + + PutWarmerResponse anotherPutWarmerResponse = client().admin().indices().preparePutWarmer("second_custom_warmer") + .setSearchRequest(client().prepareSearch("test").setTypes("test").setQuery(QueryBuilders.matchAllQuery())) + .execute().actionGet(); + assertThat(anotherPutWarmerResponse.isAcknowledged(), equalTo(true)); + + GetWarmersResponse getWarmersResponse = client().admin().indices().prepareGetWarmers("*").addWarmers("*").get(); + assertThat(getWarmersResponse.warmers().size(), is(1)); + + getWarmersResponse = client().admin().indices().prepareGetWarmers("_all").addWarmers("_all").get(); + assertThat(getWarmersResponse.warmers().size(), is(1)); + + getWarmersResponse = client().admin().indices().prepareGetWarmers("t*").addWarmers("c*").get(); + assertThat(getWarmersResponse.warmers().size(), is(1)); + + getWarmersResponse = client().admin().indices().prepareGetWarmers("test").addWarmers("custom_warmer", "second_custom_warmer").get(); + assertThat(getWarmersResponse.warmers().size(), is(1)); + } + private long getWarmerRuns() { IndicesStatsResponse indicesStatsResponse = client().admin().indices().prepareStats("test").clear().setWarmer(true).execute().actionGet(); return indicesStatsResponse.getIndex("test").getPrimaries().warmer.total(); From a3abcdc93a004bea53ba2d4cbab585f8820b660d Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Tue, 14 Jan 2014 22:39:28 +0100 Subject: [PATCH 19/34] Consistent APIs: Get field mapping API includes 'mappings' The get field mapping API now includes a mappings element after the index in its JSON Added more consistent endpoint /{index}/_mapping/{type}/field/{fields} and added endpoint /_mapping/{type}/field/{fields} which are also used in tests Added rest spec tests for wildcards and _all Relates #4071 NOTE: This is not yet complete for 1.0. We need to return an empty JSON document instead of a 404 if the field of an existing index and type is not found. However this is not possible with the current data structure being returned. Needs to be finished for 1.0. --- .../api/indices.get_field_mapping.json | 2 +- .../indices.get_field_mapping/10_basic.yaml | 25 +++-- .../50_field_wildcards.yaml | 104 +++++++++++++++--- .../mapping/get/GetFieldMappingsResponse.java | 2 + .../get/RestGetFieldMappingAction.java | 5 +- 5 files changed, 112 insertions(+), 26 deletions(-) diff --git a/rest-api-spec/api/indices.get_field_mapping.json b/rest-api-spec/api/indices.get_field_mapping.json index b683bf4ee0c..1d823e23104 100644 --- a/rest-api-spec/api/indices.get_field_mapping.json +++ b/rest-api-spec/api/indices.get_field_mapping.json @@ -4,7 +4,7 @@ "methods": ["GET"], "url": { "path": "/_mapping/field/{field}", - "paths": ["/_mapping/field/{field}", "/{index}/_mapping/field/{field}", "/{index}/{type}/_mapping/field/{field}"], + "paths": ["/_mapping/field/{field}", "/{index}/_mapping/field/{field}", "/_mapping/{type}/field/{field}", "/{index}/_mapping/{type}/field/{field}"], "parts": { "index": { "type" : "list", diff --git a/rest-api-spec/test/indices.get_field_mapping/10_basic.yaml b/rest-api-spec/test/indices.get_field_mapping/10_basic.yaml index 4d4614629c2..bac6fffc0a5 100644 --- a/rest-api-spec/test/indices.get_field_mapping/10_basic.yaml +++ b/rest-api-spec/test/indices.get_field_mapping/10_basic.yaml @@ -17,7 +17,7 @@ setup: indices.get_field_mapping: field: text - - match: {test_index.test_type.text.mapping.text.type: string} + - match: {test_index.mappings.test_type.text.mapping.text.type: string} --- "Get field mapping by index only": @@ -26,7 +26,7 @@ setup: index: test_index field: text - - match: {test_index.test_type.text.mapping.text.type: string} + - match: {test_index.mappings.test_type.text.mapping.text.type: string} --- "Get field mapping by type & field": @@ -37,7 +37,7 @@ setup: type: test_type field: text - - match: {test_index.test_type.text.mapping.text.type: string} + - match: {test_index.mappings.test_type.text.mapping.text.type: string} --- "Get field mapping by type & field, with another field that doesn't exist": @@ -48,8 +48,8 @@ setup: type: test_type field: [ text , text1 ] - - match: {test_index.test_type.text.mapping.text.type: string} - - is_false: test_index.test_type.text1 + - match: {test_index.mappings.test_type.text.mapping.text.type: string} + - is_false: test_index.mappings.test_type.text1 --- "Get field mapping with include_defaults": @@ -61,5 +61,16 @@ setup: field: text include_defaults: true - - match: {test_index.test_type.text.mapping.text.type: string} - - match: {test_index.test_type.text.mapping.text.analyzer: default} + - match: {test_index.mappings.test_type.text.mapping.text.type: string} + - match: {test_index.mappings.test_type.text.mapping.text.analyzer: default} + +--- +"Get field mapping should work without index specifying type and field": + + - do: + indices.get_field_mapping: + type: test_type + field: text + + - match: {test_index.mappings.test_type.text.mapping.text.type: string} + diff --git a/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml b/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml index 879d74988f5..fcf59263266 100644 --- a/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml +++ b/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml @@ -23,6 +23,29 @@ setup: type: string index_name: t3 + - do: + indices.create: + index: test_index_2 + body: + mappings: + test_type_2: + properties: + t1: + type: string + t2: + type: string + obj: + path: just_name + properties: + t1: + type: string + i_t1: + type: string + index_name: t1 + i_t3: + type: string + index_name: t3 + --- "Get field mapping with * for fields": @@ -30,43 +53,92 @@ setup: indices.get_field_mapping: field: "*" - - match: {test_index.test_type.t1.full_name: t1 } - - match: {test_index.test_type.t2.full_name: t2 } - - match: {test_index.test_type.obj\.t1.full_name: obj.t1 } - - match: {test_index.test_type.obj\.i_t1.full_name: obj.i_t1 } - - match: {test_index.test_type.obj\.i_t3.full_name: obj.i_t3 } + - match: {test_index.mappings.test_type.t1.full_name: t1 } + - match: {test_index.mappings.test_type.t2.full_name: t2 } + - match: {test_index.mappings.test_type.obj\.t1.full_name: obj.t1 } + - match: {test_index.mappings.test_type.obj\.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.obj\.i_t3.full_name: obj.i_t3 } --- "Get field mapping with t* for fields": - do: indices.get_field_mapping: + index: test_index field: "t*" # i_t1 matches the pattern using it's index name, but t1 already means a full name # of a field and thus takes precedence. - - match: {test_index.test_type.t1.full_name: t1 } - - match: {test_index.test_type.t2.full_name: t2 } - - match: {test_index.test_type.t3.full_name: obj.i_t3 } - - length: {test_index.test_type: 3} + - match: {test_index.mappings.test_type.t1.full_name: t1 } + - match: {test_index.mappings.test_type.t2.full_name: t2 } + - match: {test_index.mappings.test_type.t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 3} --- "Get field mapping with *t1 for fields": - do: indices.get_field_mapping: + index: test_index field: "*t1" - - match: {test_index.test_type.t1.full_name: t1 } - - match: {test_index.test_type.obj\.t1.full_name: obj.t1 } - - match: {test_index.test_type.obj\.i_t1.full_name: obj.i_t1 } - - length: {test_index.test_type: 3} + - match: {test_index.mappings.test_type.t1.full_name: t1 } + - match: {test_index.mappings.test_type.obj\.t1.full_name: obj.t1 } + - match: {test_index.mappings.test_type.obj\.i_t1.full_name: obj.i_t1 } + - length: {test_index.mappings.test_type: 3} --- "Get field mapping with wildcarded relative names": - do: indices.get_field_mapping: + index: test_index field: "i_*" - - match: {test_index.test_type.i_t1.full_name: obj.i_t1 } - - match: {test_index.test_type.i_t3.full_name: obj.i_t3 } - - length: {test_index.test_type: 2} + - match: {test_index.mappings.test_type.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.i_t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 2} + +--- +"Get field mapping should work using '_all' for indices and types": + + - do: + indices.get_field_mapping: + index: _all + type: _all + field: "i_*" + - match: {test_index.mappings.test_type.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.i_t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 2} + - match: {test_index_2.mappings.test_type_2.i_t1.full_name: obj.i_t1 } + - match: {test_index_2.mappings.test_type_2.i_t3.full_name: obj.i_t3 } + - length: {test_index_2.mappings.test_type_2: 2} + +--- +"Get field mapping should work using '*' for indices and types": + + - do: + indices.get_field_mapping: + index: '*' + type: '*' + field: "i_*" + - match: {test_index.mappings.test_type.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.i_t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 2} + - match: {test_index_2.mappings.test_type_2.i_t1.full_name: obj.i_t1 } + - match: {test_index_2.mappings.test_type_2.i_t3.full_name: obj.i_t3 } + - length: {test_index_2.mappings.test_type_2: 2} + +--- +"Get field mapping should work using comma_separated values for indices and types": + + - do: + indices.get_field_mapping: + index: 'test_index,test_index_2' + type: 'test_type,test_type_2' + field: "i_*" + - match: {test_index.mappings.test_type.i_t1.full_name: obj.i_t1 } + - match: {test_index.mappings.test_type.i_t3.full_name: obj.i_t3 } + - length: {test_index.mappings.test_type: 2} + - match: {test_index_2.mappings.test_type_2.i_t1.full_name: obj.i_t1 } + - match: {test_index_2.mappings.test_type_2.i_t3.full_name: obj.i_t3 } + - length: {test_index_2.mappings.test_type_2: 2} + diff --git a/src/main/java/org/elasticsearch/action/admin/indices/mapping/get/GetFieldMappingsResponse.java b/src/main/java/org/elasticsearch/action/admin/indices/mapping/get/GetFieldMappingsResponse.java index da612f602d9..a62ea0d353b 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/mapping/get/GetFieldMappingsResponse.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/mapping/get/GetFieldMappingsResponse.java @@ -71,6 +71,7 @@ public class GetFieldMappingsResponse extends ActionResponse implements ToXConte public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { for (Map.Entry>> indexEntry : mappings.entrySet()) { builder.startObject(indexEntry.getKey(), XContentBuilder.FieldCaseConversion.NONE); + builder.startObject("mappings"); for (Map.Entry> typeEntry : indexEntry.getValue().entrySet()) { builder.startObject(typeEntry.getKey(), XContentBuilder.FieldCaseConversion.NONE); for (Map.Entry fieldEntry : typeEntry.getValue().entrySet()) { @@ -81,6 +82,7 @@ public class GetFieldMappingsResponse extends ActionResponse implements ToXConte builder.endObject(); } builder.endObject(); + builder.endObject(); } return builder; } diff --git a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java index 946f29a7e40..269098d0a87 100644 --- a/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java +++ b/src/main/java/org/elasticsearch/rest/action/admin/indices/mapping/get/RestGetFieldMappingAction.java @@ -49,17 +49,18 @@ public class RestGetFieldMappingAction extends BaseRestHandler { public RestGetFieldMappingAction(Settings settings, Client client, RestController controller) { super(settings, client); controller.registerHandler(GET, "/_mapping/field/{fields}", this); + controller.registerHandler(GET, "/_mapping/{type}/field/{fields}", this); controller.registerHandler(GET, "/{index}/_mapping/field/{fields}", this); controller.registerHandler(GET, "/{index}/{type}/_mapping/field/{fields}", this); + controller.registerHandler(GET, "/{index}/_mapping/{type}/field/{fields}", this); } @Override public void handleRequest(final RestRequest request, final RestChannel channel) { final String[] indices = Strings.splitStringByCommaToArray(request.param("index")); - final String[] types = Strings.splitStringByCommaToArray(request.param("type")); + final String[] types = request.paramAsStringArrayOrEmptyIfAll("type"); boolean local = request.paramAsBooleanOptional("local", false); final String[] fields = Strings.splitStringByCommaToArray(request.param("fields")); - GetFieldMappingsRequest getMappingsRequest = new GetFieldMappingsRequest(); getMappingsRequest.indices(indices).types(types).local(local).fields(fields).includeDefaults(request.paramAsBoolean("include_defaults", false)); getMappingsRequest.indicesOptions(IndicesOptions.fromRequest(request, getMappingsRequest.indicesOptions())); From 6389432b28817b82b0ae8d5c67f0f7a47c32cbaa Mon Sep 17 00:00:00 2001 From: Britta Weber Date: Tue, 14 Jan 2014 22:25:15 +0100 Subject: [PATCH 20/34] Use most recent cluster state to find types in delete mapping Previously, the cluster state before flushing was used to check which types map the given types pattern. However, this state might not be up to date. Instead use the more recent cluster state from clusterState.state() This fixes a test failure of PercolatorTests.testDeletePercolatorType Other changes: - use BoolFilter instead of OrFilter, because it is faster - throw exception immediately when no type matching the given patterns was found in cluster state --- .../delete/TransportDeleteMappingAction.java | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/TransportDeleteMappingAction.java b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/TransportDeleteMappingAction.java index 113aba2a97a..f55e7149cb4 100644 --- a/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/TransportDeleteMappingAction.java +++ b/src/main/java/org/elasticsearch/action/admin/indices/mapping/delete/TransportDeleteMappingAction.java @@ -43,9 +43,11 @@ import org.elasticsearch.cluster.metadata.MetaDataMappingService; import org.elasticsearch.common.collect.ImmutableOpenMap; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; -import org.elasticsearch.index.query.OrFilterBuilder; +import org.elasticsearch.index.Index; +import org.elasticsearch.index.query.BoolFilterBuilder; import org.elasticsearch.index.query.QueryBuilders; import org.elasticsearch.index.query.TypeFilterBuilder; +import org.elasticsearch.indices.TypeMissingException; import org.elasticsearch.node.settings.NodeSettingsService; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.transport.TransportService; @@ -117,18 +119,21 @@ public class TransportDeleteMappingAction extends TransportMasterNodeOperationAc public void onResponse(FlushResponse flushResponse) { // get all types that need to be deleted. - ImmutableOpenMap> result = state.metaData().findMappings( + ImmutableOpenMap> result = clusterService.state().metaData().findMappings( request.indices(), request.types() ); // create OrFilter with type filters within to account for different types - OrFilterBuilder filterBuilder = new OrFilterBuilder(); + BoolFilterBuilder filterBuilder = new BoolFilterBuilder(); Set types = new HashSet(); for (ObjectObjectCursor> typesMeta : result) { for (ObjectObjectCursor type : typesMeta.value) { - filterBuilder.add(new TypeFilterBuilder(type.key)); + filterBuilder.should(new TypeFilterBuilder(type.key)); types.add(type.key); } } + if (types.size() == 0) { + throw new TypeMissingException(new Index("_all"), request.types(), "No index has the type."); + } request.types(types.toArray(new String[types.size()])); QuerySourceBuilder querySourceBuilder = new QuerySourceBuilder() .setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), filterBuilder)); From 2f115b81035e76a928a4cf6629eb02fbc5175761 Mon Sep 17 00:00:00 2001 From: Britta Weber Date: Wed, 15 Jan 2014 00:22:44 +0100 Subject: [PATCH 21/34] update rest spec to be consistent with recent changes see issue #4071 --- rest-api-spec/api/indices.delete_alias.json | 8 ++++---- rest-api-spec/api/indices.delete_mapping.json | 6 +++--- rest-api-spec/api/indices.delete_warmer.json | 12 ++++++------ rest-api-spec/api/indices.put_alias.json | 5 ++--- rest-api-spec/api/indices.put_mapping.json | 3 +-- rest-api-spec/api/indices.put_warmer.json | 3 +-- 6 files changed, 17 insertions(+), 20 deletions(-) diff --git a/rest-api-spec/api/indices.delete_alias.json b/rest-api-spec/api/indices.delete_alias.json index 03686046820..532298866c0 100644 --- a/rest-api-spec/api/indices.delete_alias.json +++ b/rest-api-spec/api/indices.delete_alias.json @@ -7,14 +7,14 @@ "paths": ["/{index}/_alias/{name}", "/{index}/_aliases/{name}"], "parts": { "index": { - "type" : "string", + "type" : "list", "required" : true, - "description" : "The name of the index with an alias" + "description" : "A comma-separated list of index names (supports wildcards); use `_all` for all indices" }, "name": { - "type" : "string", + "type" : "list", "required" : true, - "description" : "The name of the alias to be deleted" + "description" : "A comma-separated list of aliases to delete (supports wildcards); use `_all` to delete all aliases for the specified indices." } }, "params": { diff --git a/rest-api-spec/api/indices.delete_mapping.json b/rest-api-spec/api/indices.delete_mapping.json index 6494b3231d9..3fbe66c0a65 100644 --- a/rest-api-spec/api/indices.delete_mapping.json +++ b/rest-api-spec/api/indices.delete_mapping.json @@ -9,12 +9,12 @@ "index": { "type" : "list", "required" : true, - "description" : "A comma-separated list of index names; use `_all` for all indices" + "description" : "A comma-separated list of index names (supports wildcards); use `_all` for all indices" }, "type": { - "type" : "string", + "type" : "list", "required" : true, - "description" : "The name of the document type to delete" + "description" : "A comma-separated list of document types to delete (supports wildcards); use `_all` to delete all document types in the specified indices." } }, "params": { diff --git a/rest-api-spec/api/indices.delete_warmer.json b/rest-api-spec/api/indices.delete_warmer.json index be747cbeeec..30357befe02 100644 --- a/rest-api-spec/api/indices.delete_warmer.json +++ b/rest-api-spec/api/indices.delete_warmer.json @@ -9,21 +9,21 @@ "index": { "type" : "list", "required" : true, - "description" : "A comma-separated list of index names to register warmer for; use `_all` or empty string to perform the operation on all indices" + "description" : "A comma-separated list of index names to delete warmers from (supports wildcards); use `_all` to perform the operation on all indices." }, "name" : { - "type" : "string", - "description" : "The name of the warmer (supports wildcards); leave empty to delete all warmers" - }, - "type": { "type" : "list", - "description" : "A comma-separated list of document types to register warmer for; use `_all` or empty string to perform the operation on all types" + "description" : "A comma-separated list of warmer names to delete (supports wildcards); use `_all` to delete all warmers in the specified indices. You must specify a name either in the uri or in the parameters." } }, "params": { "master_timeout": { "type" : "time", "description" : "Specify timeout for connection to master" + }, + "name" : { + "type" : "list", + "description" : "A comma-separated list of warmer names to delete (supports wildcards); use `_all` to delete all warmers in the specified indices. You must specify a name either in the uri or in the parameters." } } }, diff --git a/rest-api-spec/api/indices.put_alias.json b/rest-api-spec/api/indices.put_alias.json index db1072e4720..cb3172a49c6 100644 --- a/rest-api-spec/api/indices.put_alias.json +++ b/rest-api-spec/api/indices.put_alias.json @@ -7,9 +7,8 @@ "paths": ["/{index}/_alias/{name}", "/_alias/{name}", "/{index}/_aliases/{name}", "/_aliases/{name}"], "parts": { "index": { - "type" : "string", - "required" : true, - "description" : "The name of the index with an alias" + "type" : "list", + "description" : "A comma-separated list of index names the alias should point to (supports wildcards); use `_all` or omit to perform the operation on all indices." }, "name": { "type" : "string", diff --git a/rest-api-spec/api/indices.put_mapping.json b/rest-api-spec/api/indices.put_mapping.json index eadba64f3ce..09d298e0764 100644 --- a/rest-api-spec/api/indices.put_mapping.json +++ b/rest-api-spec/api/indices.put_mapping.json @@ -8,8 +8,7 @@ "parts": { "index": { "type" : "list", - "required" : false, - "description" : "A comma-separated list of index names; use `_all` to perform the operation on all indices" + "description" : "A comma-separated list of index names the mapping should be added to (supports wildcards); use `_all` or omit to add the mapping on all indices." }, "type": { "type" : "string", diff --git a/rest-api-spec/api/indices.put_warmer.json b/rest-api-spec/api/indices.put_warmer.json index dedb425c585..70ca77513e6 100644 --- a/rest-api-spec/api/indices.put_warmer.json +++ b/rest-api-spec/api/indices.put_warmer.json @@ -8,8 +8,7 @@ "parts": { "index": { "type" : "list", - "required" : true, - "description" : "A comma-separated list of index names to register the warmer for; use `_all` or empty string to perform the operation on all indices" + "description" : "A comma-separated list of index names to register the warmer for; use `_all` or omit to perform the operation on all indices" }, "name": { "type" : "string", From f8a427e2668094e3392994275d659cc8add72900 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 14:00:08 +0100 Subject: [PATCH 22/34] [DOCS] Moved fielddata circuit breaker higher up the page --- .../index-modules/fielddata.asciidoc | 64 ++++++++++--------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/docs/reference/index-modules/fielddata.asciidoc b/docs/reference/index-modules/fielddata.asciidoc index 3c962ea49b5..1c382c662c3 100644 --- a/docs/reference/index-modules/fielddata.asciidoc +++ b/docs/reference/index-modules/fielddata.asciidoc @@ -24,7 +24,39 @@ field data after a certain time of inactivity. Defaults to `-1`. For example, can be set to `5m` for a 5 minute expiry. |======================================================================= -=== Field data formats +[float] +[[fielddata-circuit-breaker]] +=== Field data circuit breaker +The field data circuit breaker allows Elasticsearch to estimate the amount of +memory a field will required to be loaded into memory. It can then prevent the +field data loading by raising and exception. By default it is configured with +no limit (-1 bytes), but is automatically set to `indices.fielddata.cache.size` +if set. It can be configured with the following parameters: + +[cols="<,<",options="header",] +|======================================================================= +|Setting |Description +|`indices.fielddata.breaker.limit` |Maximum size of estimated field data +to allow loading. Defaults to 80% of the maximum JVM heap. +|`indices.fielddata.breaker.overhead` |A constant that all field data +estimations are multiplied with to determine a final estimation. Defaults to +1.03 +|======================================================================= + +Both the `indices.fielddata.breaker.limit` and +`indices.fielddata.breaker.overhead` can be changed dynamically using the +cluster update settings API. + +[float] +[[fielddata-monitoring]] +=== Monitoring field data + +You can monitor memory usage for field data as well as the field data circuit +breaker using +<> + +[[fielddata-formats]] +== Field data formats The field data format controls how field data should be stored. @@ -237,33 +269,3 @@ The `frequency` and `regex` filters can be combined: } -------------------------------------------------- -[float] -[[field-data-circuit-breaker]] -=== Field data circuit breaker -The field data circuit breaker allows Elasticsearch to estimate the amount of -memory a field will required to be loaded into memory. It can then prevent the -field data loading by raising and exception. By default it is configured with -no limit (-1 bytes), but is automatically set to `indices.fielddata.cache.size` -if set. It can be configured with the following parameters: - -[cols="<,<",options="header",] -|======================================================================= -|Setting |Description -|`indices.fielddata.breaker.limit` |Maximum size of estimated field data -to allow loading. Defaults to 80% of the maximum JVM heap. -|`indices.fielddata.breaker.overhead` |A constant that all field data -estimations are multiplied with to determine a final estimation. Defaults to -1.03 -|======================================================================= - -Both the `indices.fielddata.breaker.limit` and -`indices.fielddata.breaker.overhead` can be changed dynamically using the -cluster update settings API. - -[float] -[[field-data-monitoring]] -=== Monitoring field data - -You can monitor memory usage for field data as well as the field data circuit -breaker using -<> From a0b993e2dc197582668dc4dbbe8e64eb1f4ba53a Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 14:51:18 +0100 Subject: [PATCH 23/34] [DOCS] Tidied up cluster settings docs --- docs/reference/modules/cluster.asciidoc | 65 ++++++++++--------------- 1 file changed, 26 insertions(+), 39 deletions(-) diff --git a/docs/reference/modules/cluster.asciidoc b/docs/reference/modules/cluster.asciidoc index 9d0eefe3076..aaad74d62d8 100644 --- a/docs/reference/modules/cluster.asciidoc +++ b/docs/reference/modules/cluster.asciidoc @@ -12,70 +12,57 @@ handling nodes being added or removed. The following settings may be used: `cluster.routing.allocation.allow_rebalance`:: - Allow to control when rebalancing will happen based on the total - state of all the indices shards in the cluster. `always`, - `indices_primaries_active`, and `indices_all_active` are allowed, - defaulting to `indices_all_active` to reduce chatter during + Allow to control when rebalancing will happen based on the total + state of all the indices shards in the cluster. `always`, + `indices_primaries_active`, and `indices_all_active` are allowed, + defaulting to `indices_all_active` to reduce chatter during initial recovery. `cluster.routing.allocation.cluster_concurrent_rebalance`:: - Allow to control how many concurrent rebalancing of shards are + Allow to control how many concurrent rebalancing of shards are allowed cluster wide, and default it to `2`. `cluster.routing.allocation.node_initial_primaries_recoveries`:: - Allow to control specifically the number of initial recoveries - of primaries that are allowed per node. Since most times local - gateway is used, those should be fast and we can handle more of + Allow to control specifically the number of initial recoveries + of primaries that are allowed per node. Since most times local + gateway is used, those should be fast and we can handle more of those per node without creating load. `cluster.routing.allocation.node_concurrent_recoveries`:: - How many concurrent recoveries are allowed to happen on a node. + How many concurrent recoveries are allowed to happen on a node. Defaults to `2`. -added[1.0.0.RC1] - `cluster.routing.allocation.enable`:: Controls shard allocation for all indices, by allowing specific - kinds of shard to be allocated. Can be set to: + kinds of shard to be allocated. + added[1.0.0.RC1,Replaces `cluster.routing.allocation.disable*`] + Can be set to: * `all` (default) - Allows shard allocation for all kinds of shards. * `primaries` - Allows shard allocation only for primary shards. * `new_primaries` - Allows shard allocation only for primary shards for new indices. * `none` - No shard allocations of any kind are allowed for all indices. `cluster.routing.allocation.disable_new_allocation`:: - Allows to disable new primary allocations. Note, this will prevent - allocations for newly created indices. This setting really make - sense when dynamically updating it using the cluster update - settings API. This setting has been deprecated in favour - for `cluster.routing.allocation.enable`. - + deprecated[1.0.0.RC1,Replaced by `cluster.routing.allocation.enable`] `cluster.routing.allocation.disable_allocation`:: - Allows to disable either primary or replica allocation (does not - apply to newly created primaries, see `disable_new_allocation` - above). Note, a replica will still be promoted to primary if - one does not exist. This setting really make sense when - dynamically updating it using the cluster update settings API. - This setting has been deprecated in favour for `cluster.routing.allocation.enable`. + deprecated[1.0.0.RC1,Replaced by `cluster.routing.allocation.enable`] `cluster.routing.allocation.disable_replica_allocation`:: - Allows to disable only replica allocation. Similar to the previous - setting, mainly make sense when using it dynamically using the - cluster update settings API. This setting has been deprecated in - favour for `cluster.routing.allocation.enable`. + deprecated[1.0.0.RC1,Replaced by `cluster.routing.allocation.enable`] `cluster.routing.allocation.same_shard.host`:: Prevents that multiple instances of the same shard are allocated - on a single host. Defaults to `false`. This setting only applies + on a single host. Defaults to `false`. This setting only applies if multiple nodes are started on the same machine. `indices.recovery.concurrent_streams`:: - The number of streams to open (on a *node* level) to recover a - shard from a peer shard. Defaults to `3`. + The number of streams to open (on a *node* level) to recover a + shard from a peer shard. Defaults to `3`. [float] [[allocation-awareness]] @@ -182,8 +169,8 @@ set to `value1` and `value2` by setting [source,js] -------------------------------------------------- curl -XPUT localhost:9200/test/_settings -d '{ - "index.routing.allocation.include.tag" : "value1,value2" -}' + "index.routing.allocation.include.tag" : "value1,value2" +}' -------------------------------------------------- On the other hand, we can create an index that will be deployed on all @@ -193,11 +180,11 @@ nodes except for nodes with a `tag` of value `value3` by setting [source,js] -------------------------------------------------- curl -XPUT localhost:9200/test/_settings -d '{ - "index.routing.allocation.exclude.tag" : "value3" -}' + "index.routing.allocation.exclude.tag" : "value3" +}' -------------------------------------------------- -`index.routing.allocation.require.*` can be used to +`index.routing.allocation.require.*` can be used to specify a number of rules, all of which MUST match in order for a shard to be allocated to a node. This is in contrast to `include` which will include a node if ANY rule matches. @@ -229,7 +216,7 @@ curl -XPUT localhost:9200/test/_settings -d '{ "index.routing.allocation.include.group2" : "yyy", "index.routing.allocation.exclude.group3" : "zzz", "index.routing.allocation.require.group4" : "aaa" -}' +}' -------------------------------------------------- The provided settings can also be updated in real time using the update @@ -246,6 +233,6 @@ address: curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "cluster.routing.allocation.exclude._ip" : "10.0.0.1" - } -}' + } +}' -------------------------------------------------- From 353d599d05918dc847ec0753e8851fd80cc2e497 Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Wed, 15 Jan 2014 14:57:00 +0100 Subject: [PATCH 24/34] Increased max map pages for systemd to align with init.d settings in packages --- src/rpm/systemd/sysctl.d/elasticsearch.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rpm/systemd/sysctl.d/elasticsearch.conf b/src/rpm/systemd/sysctl.d/elasticsearch.conf index daf621bd23c..62ea54d8697 100644 --- a/src/rpm/systemd/sysctl.d/elasticsearch.conf +++ b/src/rpm/systemd/sysctl.d/elasticsearch.conf @@ -1 +1 @@ -vm.max_map_count=65535 +vm.max_map_count=262144 From 3062e59f51701a14c82f0dffa35120232b3dd43a Mon Sep 17 00:00:00 2001 From: Lee Hinman Date: Wed, 15 Jan 2014 07:04:24 -0700 Subject: [PATCH 25/34] [DOCS] Fix default setting in circuit breaker documentation --- docs/reference/index-modules/fielddata.asciidoc | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/reference/index-modules/fielddata.asciidoc b/docs/reference/index-modules/fielddata.asciidoc index 1c382c662c3..c958dcbc037 100644 --- a/docs/reference/index-modules/fielddata.asciidoc +++ b/docs/reference/index-modules/fielddata.asciidoc @@ -29,9 +29,9 @@ example, can be set to `5m` for a 5 minute expiry. === Field data circuit breaker The field data circuit breaker allows Elasticsearch to estimate the amount of memory a field will required to be loaded into memory. It can then prevent the -field data loading by raising and exception. By default it is configured with -no limit (-1 bytes), but is automatically set to `indices.fielddata.cache.size` -if set. It can be configured with the following parameters: +field data loading by raising and exception. By default the limit is configured +to 80% of the maximum JVM heap. It can be configured with the following +parameters: [cols="<,<",options="header",] |======================================================================= @@ -268,4 +268,3 @@ The `frequency` and `regex` filters can be combined: } } -------------------------------------------------- - From 93ba3b5e70a59e82dc79754fc84a54ab83dd8595 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 15:09:18 +0100 Subject: [PATCH 26/34] [DOCS] Tidied up layout of setup docs --- docs/reference/setup/as-a-service-win.asciidoc | 3 ++- docs/reference/setup/as-a-service.asciidoc | 8 +++++++- docs/reference/setup/repositories.asciidoc | 2 ++ 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/reference/setup/as-a-service-win.asciidoc b/docs/reference/setup/as-a-service-win.asciidoc index e786741b241..f77f194e6f9 100644 --- a/docs/reference/setup/as-a-service-win.asciidoc +++ b/docs/reference/setup/as-a-service-win.asciidoc @@ -1,7 +1,7 @@ [[setup-service-win]] == Running as a Service on Windows -Windows users can configure Elasticsearch to run as a service to run in the background or start automatically +Windows users can configure Elasticsearch to run as a service to run in the background or start automatically at startup without any user interaction. This can be achieved through `service.bat` script under `bin/` folder which allows one to install, remove, manage or configure the service and potentially start and stop the service, all from the command-line. @@ -47,6 +47,7 @@ The service 'elasticsearch-service-x64' has been installed. NOTE: While a JRE can be used for the Elasticsearch service, due to its use of a client VM (as oppose to a server JVM which offers better performance for long-running applications) its usage is discouraged and a warning will be issued. +[float] === Customizing service settings There are two ways to customize the service settings: diff --git a/docs/reference/setup/as-a-service.asciidoc b/docs/reference/setup/as-a-service.asciidoc index f0e323385b8..59f2163b92c 100644 --- a/docs/reference/setup/as-a-service.asciidoc +++ b/docs/reference/setup/as-a-service.asciidoc @@ -3,6 +3,7 @@ In order to run elasticsearch as a service on your operating system, the provided packages try to make it as easy as possible for you to start and stop elasticsearch during reboot and upgrades. +[float] === Linux Currently our build automatically creates a debian package and an RPM package, which is available on the download page. The package itself does not have any dependencies, but you have to make sure that you installed a JDK. @@ -26,6 +27,7 @@ Each package features a configuration file, which allows you to set the followin `ES_JAVA_OPTS`:: Any additional java options you may want to apply. This may be useful, if you need to set the `node.name` property, but do not want to change the `elasticsearch.yml` configuration file, because it is distributed via a provisioning system like puppet or chef. Example: `ES_JAVA_OPTS="-Des.node.name=search-01"` `RESTART_ON_UPGRADE`:: Configure restart on package upgrade, defaults to `false`. This means you will have to restart your elasticsearch instance after installing a package manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continouos shard reallocation resulting in high network traffic and reducing the response times of your cluster. +[float] ==== Debian/Ubuntu The debian package ships with everything you need as it uses standard debian tools like update `update-rc.d` to define the runlevels it runs on. The init script is placed at `/etc/init.d/elasticsearch` is you would expect it. The configuration file is placed at `/etc/default/elasticsearch`. @@ -38,6 +40,7 @@ sudo update-rc.d elasticsearch defaults 95 10 sudo /etc/init.d/elasticsearch start -------------------------------------------------- +[float] ===== Installing the oracle JDK The usual recommendation is to run the Oracle JDK with elasticsearch. However Ubuntu and Debian only ship the OpenJDK due to license issues. You can easily install the oracle installer package though. In case you are missing the `add-apt-repository` command under Debian GNU/Linux, make sure have at least Debian Wheezy and the package `python-software-properties` installed @@ -53,8 +56,10 @@ java -version The last command should verify a successful installation of the Oracle JDK. -==== RPM based distributions +[float] +==== RPM based distributions +[float] ===== Using chkconfig Some RPM based distributions are using `chkconfig` to enable and disable services. The init script is located at `/etc/init.d/elasticsearch`, where as the configuration file is placed at `/etc/sysconfig/elasticsearch`. Like the debian package the RPM package is not started by default after installation, you have to do this manually by entering the following commands @@ -66,6 +71,7 @@ sudo service elasticsearch start -------------------------------------------------- +[float] ===== Using systemd Distributions like SuSe do not use the `chkconfig` tool to register services, but rather `systemd` and its command `/bin/systemctl` to start and stop services (at least in newer versions, otherwise use the `chkconfig` commands above). The configuration file is also placed at `/etc/sysconfig/elasticsearch`. After installing the RPM, you have to change the systemd configuration and then start up elasticsearch diff --git a/docs/reference/setup/repositories.asciidoc b/docs/reference/setup/repositories.asciidoc index 838e3e33ed5..aa0ae30e7b9 100644 --- a/docs/reference/setup/repositories.asciidoc +++ b/docs/reference/setup/repositories.asciidoc @@ -6,6 +6,7 @@ We also have repositories available for APT and YUM based distributions. We have split the major versions in separate urls to avoid accidental upgrades across major version. For all 0.90.x releases use 0.90 as version number, for 1.0.x use 1.0, etc. +[float] === APT Download and install the Public Signing Key @@ -25,6 +26,7 @@ deb http://packages.elasticsearch.org/elasticsearch/0.90/debian stable main Run apt-get update and the repository is ready for use. +[float] === YUM Download and install the Public Signing Key From 12a095d797ce31584652671ea4393e1586331ac3 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 16:13:38 +0100 Subject: [PATCH 27/34] [DOCS] Tidied up the multi-indices docs --- docs/reference/api-conventions.asciidoc | 33 +++++++++++++++++-------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/docs/reference/api-conventions.asciidoc b/docs/reference/api-conventions.asciidoc index 091393ec1ae..b9e9516e514 100644 --- a/docs/reference/api-conventions.asciidoc +++ b/docs/reference/api-conventions.asciidoc @@ -26,17 +26,30 @@ support wildcards, for example: `test*`, and the ability to "add" (`+`) and "remove" (`-`), for example: `+test*,-test3`. All multi indices API support the following url query string parameters: -* `ignore_unavailable` - Controls whether to ignore if any specified indices are unavailable, this includes indices - that don't exist or closed indices. Either `true` or `false` can be specified. -* `allow_no_indices` - Controls whether to fail if a wildcard indices expressions results into no concrete indices. - Either `true` or `false` can be specified. For example if the wildcard expression `foo*` is specified and no indices - are available that start with `foo` then depending on this setting the request will fail. This setting is also applicable - when `_all`, `*` or no index has been specified. -* `expand_wildcards` - Controls to what kind of concrete indices wildcard indices expression expand to. If `open` is - specified then the wildcard expression if expanded to only open indices and if `closed` is specified then the wildcard - expression if expanded only to closed indices. Also both values (`open,closed`) can be specified to expand to all indices. -The defaults settings for the above parameters dependent on the api being used. +`ignore_unavailable`:: + +Controls whether to ignore if any specified indices are unavailable, this +includes indices that don't exist or closed indices. Either `true` or `false` +can be specified. + +`allow_no_indices`:: + +Controls whether to fail if a wildcard indices expressions results into no +concrete indices. Either `true` or `false` can be specified. For example if +the wildcard expression `foo*` is specified and no indices are available that +start with `foo` then depending on this setting the request will fail. This +setting is also applicable when `_all`, `*` or no index has been specified. + +`expand_wildcards`:: + +Controls to what kind of concrete indices wildcard indices expression expand +to. If `open` is specified then the wildcard expression if expanded to only +open indices and if `closed` is specified then the wildcard expression if +expanded only to closed indices. Also both values (`open,closed`) can be +specified to expand to all indices. + +The defaults settings for the above parameters depend on the api being used. NOTE: Single index APIs such as the <> and the <> do not support multiple indices. From 4d68d722a13e14b6c548939a36fdcc56f85e3d9b Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Wed, 15 Jan 2014 17:03:30 +0100 Subject: [PATCH 28/34] Added waiting for yellow cluster state in rest test In order to prevent rare timing issue, where the creation of an index happens in the cluster state too late and thus results in empty mappings. --- .../test/indices.get_field_mapping/50_field_wildcards.yaml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml b/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml index fcf59263266..766be059d78 100644 --- a/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml +++ b/rest-api-spec/test/indices.get_field_mapping/50_field_wildcards.yaml @@ -46,6 +46,10 @@ setup: type: string index_name: t3 + - do: + cluster.health: + wait_for_status: yellow + --- "Get field mapping with * for fields": From baf3bd92240a05e6b8d37375b8be38719cd5716e Mon Sep 17 00:00:00 2001 From: David Pilato Date: Wed, 15 Jan 2014 17:10:11 +0100 Subject: [PATCH 29/34] Don't fail plugin manager test when downloading an actual plugin --- .../java/org/elasticsearch/plugin/PluginManagerTests.java | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/test/java/org/elasticsearch/plugin/PluginManagerTests.java b/src/test/java/org/elasticsearch/plugin/PluginManagerTests.java index 65f1032b18a..1cb1dea67b1 100644 --- a/src/test/java/org/elasticsearch/plugin/PluginManagerTests.java +++ b/src/test/java/org/elasticsearch/plugin/PluginManagerTests.java @@ -227,11 +227,9 @@ public class PluginManagerTests extends ElasticsearchIntegrationTest { assertThat(plugins, notNullValue()); assertThat(plugins.length, is(0)); } catch (IOException e) { - logger.warn("--> IOException raised while downloading plugin [{}].", e, pluginShortName); - throw e; + logger.warn("--> IOException raised while downloading plugin [{}]. Skipping test.", e, pluginShortName); } catch (ElasticsearchTimeoutException e) { logger.warn("--> timeout exception raised while downloading plugin [{}]. Skipping test.", pluginShortName); - throw e; } } From faddd66e87d8cbd2466b9e4816a6ea04a728454d Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 17:50:24 +0100 Subject: [PATCH 30/34] [DOCS] Added breaking changes in 1.0 --- docs/reference/index-modules/store.asciidoc | 1 + docs/reference/index.asciidoc | 4 + docs/reference/migration/migrate_1_0.asciidoc | 345 ++++++++++++++++++ 3 files changed, 350 insertions(+) create mode 100644 docs/reference/migration/migrate_1_0.asciidoc diff --git a/docs/reference/index-modules/store.asciidoc b/docs/reference/index-modules/store.asciidoc index 247b36451ac..8388ee2d5df 100644 --- a/docs/reference/index-modules/store.asciidoc +++ b/docs/reference/index-modules/store.asciidoc @@ -81,6 +81,7 @@ Lucene `NIOFSDirectory`) using NIO. It allows multiple threads to read from the same file concurrently. It is not recommended on Windows because of a bug in the SUN Java implementation. +[[mmapfs]] [float] ==== MMap FS diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index fc2cb541813..1ae7a20dc4f 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -3,6 +3,8 @@ include::setup.asciidoc[] +include::migration/migrate_1_0.asciidoc[] + include::api-conventions.asciidoc[] include::docs.asciidoc[] @@ -29,3 +31,5 @@ include::testing.asciidoc[] include::glossary.asciidoc[] + + diff --git a/docs/reference/migration/migrate_1_0.asciidoc b/docs/reference/migration/migrate_1_0.asciidoc new file mode 100644 index 00000000000..c74f7eafed0 --- /dev/null +++ b/docs/reference/migration/migrate_1_0.asciidoc @@ -0,0 +1,345 @@ +[[breaking-changes]] += Breaking changes in 1.0 + +This section discusses the changes that you need to be aware of when migrating +your application to Elasticsearch 1.0: + +== System and settings + +* Elasticsearch now runs in the foreground by default. There is no more `-f` + flag on the command line. Instead, to run elasticsearch as a daemon, use + the `-d` flag: + +[source,sh] +--------------- +./bin/elasticsearch -d +--------------- + +* Command line settings can now be passed without the `-Des.` prefix, for + instance: + +[source,sh] +--------------- +./bin/elasticsearch --node.name=search_1 --cluster.name=production +--------------- + +* Elasticsearch on 64 bit Linux now uses <> by default. Make + sure that you set <> to a sufficiently high + number. The RPM and Debian packages default this value to `262144`. + +* The RPM and Debian packages no longer start Elasticsearch by default. + +* The `cluster.routing.allocation` settings (`disable_allocation`, + `disable_new_allocation` and `disable_replica_location`) have been + <>: + +[source,yaml] +--------------- +cluster.routing.allocation.enable: all|primaries|new_primaries|none +--------------- + +== Stats and Info APIs + +The <>, <>, +<> and <> +APIs have all been changed to make their format more RESTful and less clumsy. + +For instance, if you just want the `nodes` section of the the `cluster_state`, +instead of: + +[source,sh] +--------------- +GET /_cluster/state?filter_metadata&filter_routing_table&filter_blocks +--------------- + +you now use: + +[source,sh] +--------------- +GET /_cluster/state/nodes +--------------- + +Simliarly for the `nodes_stats` API, if you want the `transport` and `http` +metrics only, instead of: + +[source,sh] +--------------- +GET /_nodes/stats?clear&transport&http +--------------- + +you now use: + +[source,sh] +--------------- +GET /_nodes/stats/transport,http +--------------- + +See the links above for full details. + + +== Indices APIs + +The `mapping`, `alias`, `settings`, and `warmer` index APIs are all similar +but there are subtle differences in the order of the URL and the response +body. For instance, adding a mapping and a warmer look slightly different: + +[source,sh] +--------------- +PUT /{index}/{type}/_mapping +PUT /{index}/_warmer/{name} +--------------- + +These URLs have been unified as: + +[source,sh] +--------------- +PUT /{indices}/_mapping/{type} +PUT /{indices}/_alias/{name} +PUT /{indices}/_warmer/{name} + +GET /{indices}/_mapping/{types} +GET /{indices}/_alias/{names} +GET /{indices}/_settings/{names} +GET /{indices}/_warmer/{names} + +DELETE /{indices}/_mapping/{types} +DELETE /{indices}/_alias/{names} +DELETE /{indices}/_warmer/{names} +--------------- + +All of the `{indices}`, `{types}` and `{names}` parameters can be replaced by: + + * `_all`, `*` or blank (ie left out altogether), all of which mean ``all'' + * wildcards like `test*` + * comma-separated lists: `index_1,test_*` + +The only exception is `DELETE` which doesn't accept blank (missing) +parameters. If you want to delete something, you should be specific. + +Similarly, the return values for `GET` have been unified with the following +rules: + +* Only return values that exist. If you try to `GET` a mapping which doesn't + exist, then the result will be an empty object: `{}`. We no longer throw a + `404` if the requested mapping/warmer/alias/setting doesn't exist. + +* The response format always has the index name, then the section, then the + element name, for instance: + +[source,json] +--------------- +{ + "my_index": { + "mappings": { + "my_type": {...} + } + } +} +--------------- ++ +This is a breaking change for the `get_mapping` API. + +In the future we will also provide plural versions to allow putting multiple mappings etc in a single request. + +See <>, <>, <>, +<>, +<>, <>, +<>, and <> for more details. + +== Index request + +Previously a document could be indexed as itself, or wrapped in an outer +object which specified the `type` name: + +[source,json] +--------------- +PUT /my_index/my_type/1 +{ + "my_type": { + ... doc fields ... + } +} +--------------- + +This led to some ambiguity when a document also included a field with the same +name as the `type`. We no longer accept the outer `type` wrapper, but this +behaviour can be reenabled on an index-by-index basis with the setting: +`index.mapping.allow_type_wrapper`. + +== Search requests + +While the `search` API takes a top-level `query` parameter, the +<>, <> and +<> requests expected the whole body to be a +query. These have been changed to all accept a top-level `query` parameter: + +[source,json] +--------------- +GET /_count +{ + "query": { + "match": { + "title": "Interesting stuff" + } + } +} +--------------- + +Also, the top-level `filter` parameter in search has been renamed to +<>, to indicate that it should not +be used as the primary way to filter search results (use a +<> instead), but only to filter +results AFTER facets/aggregations have been calculated. + +This example counts the top colors in all matching docs, but only returns docs +with color `red`: + +[source,json] +--------------- +GET /_search +{ + "query": { + "match_all": {} + }, + "aggs": { + "colors": { + "terms": { "field": "color" } + } + }, + "post_filter": { + "term": { + "color": "red" + } + } +} +--------------- + +== Multi-fields + +Multi-fields are dead! Long live multi-fields! Well, the field type +`multi_field` has been removed. Instead, any of the core field types +(excluding `object` and `nested`) now accept a `fields` parameter. It's the +same thing, but nicer. Instead of: + +[source,json] +--------------- +"title": { + "type": "multi_field", + "fields": { + "title": { "type": "string" }, + "raw": { "type": "string", "index": "not_analyzed" } + } +} +--------------- + +you can now write: + +[source,json] +--------------- +"title": { + "type": "string"; + "fields": { + "raw": { "type": "string", "index": "not_analyzed" } + } +} +--------------- + +Existing multi-fields will be upgraded to the new format automatically. + +== Stopwords + +Previously, the <> and +<> analyzers used the list of English stopwords +by default, which caused some hard to debug indexing issues. Now they are set to +use the empty stopwords list (ie `_none_`) instead. + +== Dates without years + +When dates are specified without a year, for example: `Dec 15 10:00:00` they +are treated as dates in 2000 during indexing and range searches... except for +the upper included bound `lte` where they were treated as dates in 1970! Now, +all https://github.com/elasticsearch/elasticsearch/issues/4451[dates without years] +use `1970` as the default. + +== Parameters + +* Geo queries used to use `miles` as the default unit. And we all know what + happened at NASA because of that decision. The new default unit is + https://github.com/elasticsearch/elasticsearch/issues/4515[`meters`]. + +* For all queries that support _fuzziness_, the `min_similarity`, `fuzziness` + and `edit_distance` parameters have been unified as the single parameter + `fuzziness`. See <> for details of accepted values. + +* The `ignore_missing` parameter has been replaced by the `expand_wildcards`, + `ignore_unavailable` and `allow_no_indices` parameters, all of which have + sensible defaults. See <> for more. + +* An index name (or pattern) is now required for destructive operations like + deleting indices: + +[source,sh] +--------------- +# v0.90 - delete all indices: +DELETE / + +# v1.0 - delete all indices: +DELETE /_all +DELETE /* +--------------- ++ +Setting `action.destructive_requires_name` to `true` provides further safety +by disabling wildcard expansion on destructive actions. + +== Return values + +* The `ok` return value has been removed from all response bodies as it added + no useful information. + +* The `found`, `not_found` and `exists` return values have been unified as + `found` on all relevant APIs. + +* Field values, in response to the <> + parameter, are now always returned as arrays. A field could have single or + multiple values, which meant that sometimes they were returned as scalars + and sometimes as arrays. By always returning arrays, this simplifies user + code. The only exception to this rule is when `fields` is used to retrieve + metadata like the `routing` value, which are always singular. Metadata + fields are always returned as scalars. + +* Settings, like `index.analysis.analyzer.default` are now returned as proper + nested JSON objects, which makes them easier to work with programatically: + +[source,json] +--------------- +{ + "index": { + "analysis": { + "analyzer": { + "default": xxx + } + } + } +} +--------------- ++ +You can choose to return them in flattened format by passing `?flat_settings` +in the query string. + +* The <> API no longer supports the text response + format, but does support JSON and YAML. + +== Deprecations + +* The `text` query has been removed. Use the + <> query instead. + +* The `field` query has been removed. Use the + <> query instead. + +* Per-document boosting with the <> field has + been removed. You can use the + <> instead. + + From 9e3f527721d7f3773b16315ef35e7cbb3e747ea4 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 18:00:13 +0100 Subject: [PATCH 31/34] [DOCS] Fixed asciidoc issue --- docs/reference/migration/migrate_1_0.asciidoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/migration/migrate_1_0.asciidoc b/docs/reference/migration/migrate_1_0.asciidoc index c74f7eafed0..9972dfc1fae 100644 --- a/docs/reference/migration/migrate_1_0.asciidoc +++ b/docs/reference/migration/migrate_1_0.asciidoc @@ -1,8 +1,11 @@ [[breaking-changes]] = Breaking changes in 1.0 +[partintro] +-- This section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 1.0: +-- == System and settings From c6155c5142f7995e054f9f3b7d82f923dd3620bc Mon Sep 17 00:00:00 2001 From: Alexander Reelsen Date: Wed, 15 Jan 2014 17:02:22 +0000 Subject: [PATCH 32/34] release [1.0.0.RC1] --- docs/reference/analysis/analyzers/pattern-analyzer.asciidoc | 2 +- pom.xml | 2 +- src/main/java/org/elasticsearch/Version.java | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc b/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc index 12f8799f8f9..424ecfa766a 100644 --- a/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc +++ b/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc @@ -14,7 +14,7 @@ type: |`pattern` |The regular expression pattern, defaults to `\W+`. |`flags` |The regular expression flags. |`stopwords` |A list of stopwords to initialize the stop filter with. -Defaults to an 'empty' stopword list coming[1.0.0.RC1, Previously +Defaults to an 'empty' stopword list added[1.0.0.RC1, Previously defaulted to the English stopwords list]. Check <> for more details. |=================================================================== diff --git a/pom.xml b/pom.xml index cab82e5d666..eab765b537e 100644 --- a/pom.xml +++ b/pom.xml @@ -6,7 +6,7 @@ 4.0.0 org.elasticsearch elasticsearch - 1.0.0.RC1-SNAPSHOT + 1.0.0.RC1 jar Elasticsearch - Open Source, Distributed, RESTful Search Engine 2009 diff --git a/src/main/java/org/elasticsearch/Version.java b/src/main/java/org/elasticsearch/Version.java index b65b67e75f2..d14a9f95a23 100644 --- a/src/main/java/org/elasticsearch/Version.java +++ b/src/main/java/org/elasticsearch/Version.java @@ -148,7 +148,7 @@ public class Version implements Serializable { public static final int V_1_0_0_Beta2_ID = /*00*/1000002; public static final Version V_1_0_0_Beta2 = new Version(V_1_0_0_Beta2_ID, false, org.apache.lucene.util.Version.LUCENE_46); public static final int V_1_0_0_RC1_ID = /*00*/1000051; - public static final Version V_1_0_0_RC1 = new Version(V_1_0_0_RC1_ID, true, org.apache.lucene.util.Version.LUCENE_46); + public static final Version V_1_0_0_RC1 = new Version(V_1_0_0_RC1_ID, false, org.apache.lucene.util.Version.LUCENE_46); public static final Version CURRENT = V_1_0_0_RC1; From 3d4891321b38d19d42924566fbfc12ae1ebb1788 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Wed, 15 Jan 2014 18:22:56 +0100 Subject: [PATCH 33/34] [DOCS] Minor changes to the breaking changes doc --- docs/reference/migration/migrate_1_0.asciidoc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/reference/migration/migrate_1_0.asciidoc b/docs/reference/migration/migrate_1_0.asciidoc index 9972dfc1fae..9ea28ea2abb 100644 --- a/docs/reference/migration/migrate_1_0.asciidoc +++ b/docs/reference/migration/migrate_1_0.asciidoc @@ -4,7 +4,7 @@ [partintro] -- This section discusses the changes that you need to be aware of when migrating -your application to Elasticsearch 1.0: +your application to Elasticsearch 1.0. -- == System and settings @@ -267,8 +267,9 @@ use `1970` as the default. == Parameters -* Geo queries used to use `miles` as the default unit. And we all know what - happened at NASA because of that decision. The new default unit is +* Geo queries used to use `miles` as the default unit. And we + http://en.wikipedia.org/wiki/Mars_Climate_Orbiter[all know what + happened at NASA] because of that decision. The new default unit is https://github.com/elasticsearch/elasticsearch/issues/4515[`meters`]. * For all queries that support _fuzziness_, the `min_similarity`, `fuzziness` From 4643f7809872b86f0af033a4842658327cf6729d Mon Sep 17 00:00:00 2001 From: Igor Motov Date: Wed, 15 Jan 2014 12:57:05 -0500 Subject: [PATCH 34/34] [DOCS] Add documentation for URL repository --- docs/reference/modules/snapshots.asciidoc | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/reference/modules/snapshots.asciidoc b/docs/reference/modules/snapshots.asciidoc index 94bf40c4ab5..38db497c20f 100644 --- a/docs/reference/modules/snapshots.asciidoc +++ b/docs/reference/modules/snapshots.asciidoc @@ -71,6 +71,18 @@ on all data and master nodes. The following settings are supported: using size value notation, i.e. 1g, 10m, 5k. Defaults to `null` (unlimited chunk size). +[float] +===== Read-only URL Repository + +The URL repository (`"type": "url"`) can be used as an alternative read-only way to access data created by shared file +system repository is using shared file system to store snapshot. The URL specified in the `url` parameter should +point to the root of the shared filesystem repository. The following settings are supported: + +[horizontal] +`url`:: Location of the snapshots. Mandatory. +`concurrent_streams`:: Throttles the number of streams (per node) preforming snapshot operation. Defaults to `5` + + [float] === Snapshot @@ -170,7 +182,7 @@ http://docs.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#appendRepl ----------------------------------- $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{ "indices": "index_1,index_2", - "ignore_unavailable": "missing", + "ignore_unavailable": "true", "include_global_state": false, "rename_pattern": "index_(.)+", "rename_replacement": "restored_index_$1"