honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Allow copying source settings on resize operation (#30255) 

Today when an index is created from shrinking or splitting an existing
index, the target index inherits almost none of the source index
settings. This is surprising and a hassle for operators managing such
indices. Given this is the default behavior, we can not simply change
it. Instead, we start by introducing the ability to copy settings. This
flag can be set on the REST API or on the transport layer and it has the
behavior that it copies all settings from the source except non-copyable
settings (a property of a setting introduced in this
change). Additionally, settings on the request will always override.

This change is the first step in our adventure:
 - this flag is added here in 7.0.0 and immediately deprecated
 - this flag will be backported to 6.4.0 and remain deprecated
 - then, we will remove the ability to set this flag to false in 7.0.0
 - finally, in 8.0.0 we will remove this flag and the only behavior will
   be for settings to be copied
This commit is contained in:
Jason Tedor 2018-05-01 08:48:19 -04:00 committed by GitHub
parent 5e9e6fed90
commit 50535423ff
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
19 changed files with 577 additions and 118 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/CHANGELOG.asciidoc
View File

@ -24,6 +24,8 @@
=== Enhancements
<<copy-source-settings-on-resize, Allow copying source settings on index resize operations>> ({pull}30255[#30255])
=== Bug Fixes
Fail snapshot operations early when creating or deleting a snapshot on a repository that has been

9
docs/reference/indices/shrink-index.asciidoc
View File

@ -121,8 +121,13 @@ POST my_source_index/_shrink/my_target_index
NOTE: Mappings may not be specified in the `_shrink` request.
NOTE: By default, with the exception of `index.analysis`, `index.similarity`, and `index.sort` settings, index settings on the source
index are not copied during a shrink operation.
NOTE: By default, with the exception of `index.analysis`, `index.similarity`,
and `index.sort` settings, index settings on the source index are not copied
during a shrink operation. With the exception of non-copyable settings, settings
from the source index can be copied to the target index by adding the URL
parameter `copy_settings=true` to the request.
deprecated[7.0.0, `copy_settings` will default to `true` in 8.x and will be removed in 9.0.0]
[float]
=== Monitoring the shrink process

9
docs/reference/indices/split-index.asciidoc
View File

@ -177,8 +177,13 @@ POST my_source_index/_split/my_target_index
NOTE: Mappings may not be specified in the `_split` request.
NOTE: By default, with the exception of `index.analysis`, `index.similarity`, and `index.sort` settings, index settings on the source
index are not copied during a shrink operation.
NOTE: By default, with the exception of `index.analysis`, `index.similarity`,
and `index.sort` settings, index settings on the source index are not copied
during a split operation. With the exception of non-copyable settings, settings
from the source index can be copied to the target index by adding the URL
parameter `copy_settings=true` to the request.
deprecated[7.0.0, `copy_settings` will default to `true` in 8.x and will be removed in 9.0.0]
[float]
=== Monitoring the split process

18
docs/reference/migration/migrate_7_0/api.asciidoc
View File

@ -61,8 +61,24 @@ backwards compatibility. Backwards support for the `suggest` metric was
deprecated in 6.3.0 and now removed in 7.0.0.
[[remove-field-caps-body]]
==== In the fields capabilities API, `fields` can no longer be provided in the request body.
In the past, `fields` could be provided either as a parameter, or as part of the request
body. Specifying `fields` in the request body as opposed to a parameter was deprecated
in 6.4.0, and is now unsupported in 7.0.0.
[[copy-source-settings-on-resize]]
==== Copying source settings during shrink/split operations
In prior versions of Elasticsearch, resize operations (shrink/split) would only
copy `index.analysis`, `index.similarity`, and `index.sort` settings from the
source index. Elasticsearch 7.0.0 introduces a request parameter `copy_settings`
which will copy all index settings from the source except for non-copyable index
settings. This parameter defaults to `false` in 7.x, is immediately deprecated
in 7.0.0, will only be able to be set to `true` in 8.x, and will be removed in
9.0.0. Note than when this parameter is used it means that all copyable settings
will be copied; this includes the index blocks that must be put in place for a
resize operation, and any allocation settings put in place in preparation for
executing the resize operation. If you use this parameter, you will either have
to follow up the operation with a request to adjust to the desired settings on
the target index, or send the desired value of these settings with the resize
operation.

4
rest-api-spec/src/main/resources/rest-api-spec/api/indices.shrink.json
View File

@ -18,6 +18,10 @@
}
},
"params": {
"copy_settings": {
"type" : "boolean",
"description" : "whether or not to copy settings from the source index (defaults to false)"
},
"timeout": {
"type" : "time",
"description" : "Explicit operation timeout"

4
rest-api-spec/src/main/resources/rest-api-spec/api/indices.split.json
View File

@ -18,6 +18,10 @@
}
},
"params": {
"copy_settings": {
"type" : "boolean",
"description" : "whether or not to copy settings from the source index (defaults to false)"
},
"timeout": {
"type" : "time",
"description" : "Explicit operation timeout"

94
rest-api-spec/src/main/resources/rest-api-spec/test/indices.shrink/30_copy_settings.yml Normal file
View File

@ -0,0 +1,94 @@
---
"Copy settings during shrink index":
- skip:
version: " - 6.99.99"
reason: copy_settings did not exist prior to 7.0.0
features: "warnings"
- do:
cluster.state: {}
# get master node id
- set: { master_node: master }
- do:
indices.create:
index: source
wait_for_active_shards: 1
body:
settings:
# ensure everything is allocated on the master node
index.routing.allocation.include._id: $master
index.number_of_replicas: 0
index.merge.scheduler.max_merge_count: 4
# make it read-only
- do:
indices.put_settings:
index: source
body:
index.blocks.write: true
index.number_of_replicas: 0
- do:
cluster.health:
wait_for_status: green
index: source
# now we do a actual shrink and copy settings
- do:
indices.shrink:
index: "source"
target: "copy-settings-target"
wait_for_active_shards: 1
master_timeout: 10s
copy_settings: true
body:
settings:
index.number_of_replicas: 0
index.merge.scheduler.max_thread_count: 2
warnings:
- "parameter [copy_settings] is deprecated but was [true]"
- do:
cluster.health:
wait_for_status: green
- do:
indices.get_settings:
index: "copy-settings-target"
# settings should be copied
- match: { copy-settings-target.settings.index.merge.scheduler.max_merge_count: "4" }
- match: { copy-settings-target.settings.index.merge.scheduler.max_thread_count: "2" }
- match: { copy-settings-target.settings.index.blocks.write: "true" }
- match: { copy-settings-target.settings.index.routing.allocation.include._id: $master }
# now we do a actual shrink and do not copy settings
- do:
indices.shrink:
index: "source"
target: "no-copy-settings-target"
wait_for_active_shards: 1
master_timeout: 10s
copy_settings: false
body:
settings:
index.number_of_replicas: 0
index.merge.scheduler.max_thread_count: 2
warnings:
- "parameter [copy_settings] is deprecated but was [false]"
- do:
cluster.health:
wait_for_status: green
- do:
indices.get_settings:
index: "no-copy-settings-target"
# only the request setting should be copied
- is_false: no-copy-settings-target.settings.index.merge.scheduler.max_merge_count
- match: { no-copy-settings-target.settings.index.merge.scheduler.max_thread_count: "2" }
- is_false: no-copy-settings-target.settings.index.blocks.write
- is_false: no-copy-settings-target.settings.index.routing.allocation.include._id

98
rest-api-spec/src/main/resources/rest-api-spec/test/indices.split/30_copy_settings.yml Normal file
View File

@ -0,0 +1,98 @@
---
"Copy settings during split index":
- skip:
version: " - 6.99.99"
reason: copy_settings did not exist prior to 7.0.0
features: "warnings"
- do:
cluster.state: {}
# get master node id
- set: { master_node: master }
- do:
indices.create:
index: source
wait_for_active_shards: 1
body:
settings:
# ensure everything is allocated on the master node
index.routing.allocation.include._id: $master
index.number_of_replicas: 0
index.number_of_shards: 1
index.number_of_routing_shards: 4
index.merge.scheduler.max_merge_count: 4
# make it read-only
- do:
indices.put_settings:
index: source
body:
index.blocks.write: true
index.number_of_replicas: 0
- do:
cluster.health:
wait_for_status: green
index: source
# now we do a actual split and copy settings
- do:
indices.split:
index: "source"
target: "copy-settings-target"
wait_for_active_shards: 1
master_timeout: 10s
copy_settings: true
body:
settings:
index.number_of_replicas: 0
index.number_of_shards: 2
index.merge.scheduler.max_thread_count: 2
warnings:
- "parameter [copy_settings] is deprecated but was [true]"
- do:
cluster.health:
wait_for_status: green
- do:
indices.get_settings:
index: "copy-settings-target"
# settings should be copied
- match: { copy-settings-target.settings.index.merge.scheduler.max_merge_count: "4" }
- match: { copy-settings-target.settings.index.merge.scheduler.max_thread_count: "2" }
- match: { copy-settings-target.settings.index.blocks.write: "true" }
- match: { copy-settings-target.settings.index.routing.allocation.include._id: $master }
# now we do a actual shrink and do not copy settings
- do:
indices.split:
index: "source"
target: "no-copy-settings-target"
wait_for_active_shards: 1
master_timeout: 10s
copy_settings: false
body:
settings:
index.number_of_replicas: 0
index.number_of_shards: 2
index.merge.scheduler.max_thread_count: 2
warnings:
- "parameter [copy_settings] is deprecated but was [false]"
- do:
cluster.health:
wait_for_status: green
- do:
indices.get_settings:
index: "no-copy-settings-target"
# only the request setting should be copied
- is_false: no-copy-settings-target.settings.index.merge.scheduler.max_merge_count
- match: { no-copy-settings-target.settings.index.merge.scheduler.max_thread_count: "2" }
- is_false: no-copy-settings-target.settings.index.blocks.write
- is_false: no-copy-settings-target.settings.index.routing.allocation.include._id

11
server/src/main/java/org/elasticsearch/action/admin/indices/create/CreateIndexClusterStateUpdateRequest.java
View File

@ -45,6 +45,7 @@ public class CreateIndexClusterStateUpdateRequest extends ClusterStateUpdateRequ
private final String providedName;
private Index recoverFrom;
private ResizeType resizeType;
private boolean copySettings;
private IndexMetaData.State state = IndexMetaData.State.OPEN;
@ -112,6 +113,11 @@ public class CreateIndexClusterStateUpdateRequest extends ClusterStateUpdateRequ
return this;
}
public CreateIndexClusterStateUpdateRequest copySettings(final boolean copySettings) {
this.copySettings = copySettings;
return this;
}
public TransportMessage originalMessage() {
return originalMessage;
}
@ -170,4 +176,9 @@ public class CreateIndexClusterStateUpdateRequest extends ClusterStateUpdateRequ
public ResizeType resizeType() {
return resizeType;
}
public boolean copySettings() {
return copySettings;
}
}

18
server/src/main/java/org/elasticsearch/action/admin/indices/shrink/ResizeRequest.java
View File

@ -18,6 +18,7 @@
*/
package org.elasticsearch.action.admin.indices.shrink;
import org.elasticsearch.Version;
import org.elasticsearch.action.ActionRequestValidationException;
import org.elasticsearch.action.IndicesRequest;
import org.elasticsearch.action.admin.indices.alias.Alias;
@ -55,6 +56,7 @@ public class ResizeRequest extends AcknowledgedRequest<ResizeRequest> implements
private CreateIndexRequest targetIndexRequest;
private String sourceIndex;
private ResizeType type = ResizeType.SHRINK;
private boolean copySettings = false;
ResizeRequest() {}
@ -96,6 +98,11 @@ public class ResizeRequest extends AcknowledgedRequest<ResizeRequest> implements
} else {
type = ResizeType.SHRINK; // BWC this used to be shrink only
}
if (in.getVersion().onOrAfter(Version.V_7_0_0_alpha1)) {
copySettings = in.readBoolean();
} else {
copySettings = false;
}
}
@Override
@ -106,6 +113,9 @@ public class ResizeRequest extends AcknowledgedRequest<ResizeRequest> implements
if (out.getVersion().onOrAfter(ResizeAction.COMPATIBILITY_VERSION)) {
out.writeEnum(type);
}
if (out.getVersion().onOrAfter(Version.V_7_0_0_alpha1)) {
out.writeBoolean(copySettings);
}
}
@Override
@ -177,6 +187,14 @@ public class ResizeRequest extends AcknowledgedRequest<ResizeRequest> implements
return type;
}
public void setCopySettings(final boolean copySettings) {
this.copySettings = copySettings;
}
public boolean getCopySettings() {
return copySettings;
}
@Override
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
builder.startObject();

26
server/src/main/java/org/elasticsearch/action/admin/indices/shrink/TransportResizeAction.java
View File

@ -178,19 +178,19 @@ public class TransportResizeAction extends TransportMasterNodeAction<ResizeReque
settingsBuilder.put("index.number_of_shards", numShards);
targetIndex.settings(settingsBuilder);
return new CreateIndexClusterStateUpdateRequest(targetIndex,
cause, targetIndex.index(), targetIndexName)
// mappings are updated on the node when creating in the shards, this prevents race-conditions since all mapping must be
// applied once we took the snapshot and if somebody messes things up and switches the index read/write and adds docs we miss
// the mappings for everything is corrupted and hard to debug
.ackTimeout(targetIndex.timeout())
.masterNodeTimeout(targetIndex.masterNodeTimeout())
.settings(targetIndex.settings())
.aliases(targetIndex.aliases())
.customs(targetIndex.customs())
.waitForActiveShards(targetIndex.waitForActiveShards())
.recoverFrom(metaData.getIndex())
.resizeType(resizeRequest.getResizeType());
return new CreateIndexClusterStateUpdateRequest(targetIndex, cause, targetIndex.index(), targetIndexName)
// mappings are updated on the node when creating in the shards, this prevents race-conditions since all mapping must be
// applied once we took the snapshot and if somebody messes things up and switches the index read/write and adds docs we
// miss the mappings for everything is corrupted and hard to debug
.ackTimeout(targetIndex.timeout())
.masterNodeTimeout(targetIndex.masterNodeTimeout())
.settings(targetIndex.settings())
.aliases(targetIndex.aliases())
.customs(targetIndex.customs())
.waitForActiveShards(targetIndex.waitForActiveShards())
.recoverFrom(metaData.getIndex())
.resizeType(resizeRequest.getResizeType())
.copySettings(resizeRequest.getCopySettings());
}
@Override

73
server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataCreateIndexService.java
View File

@ -219,9 +219,19 @@ public class MetaDataCreateIndexService extends AbstractComponent {
Settings build = updatedSettingsBuilder.put(request.settings()).normalizePrefix(IndexMetaData.INDEX_SETTING_PREFIX).build();
indexScopedSettings.validate(build, true); // we do validate here - index setting must be consistent
request.settings(build);
clusterService.submitStateUpdateTask("create-index [" + request.index() + "], cause [" + request.cause() + "]",
new IndexCreationTask(logger, allocationService, request, listener, indicesService, aliasValidator, xContentRegistry, settings,
this::validate));
clusterService.submitStateUpdateTask(
"create-index [" + request.index() + "], cause [" + request.cause() + "]",
new IndexCreationTask(
logger,
allocationService,
request,
listener,
indicesService,
aliasValidator,
xContentRegistry,
settings,
this::validate,
indexScopedSettings));
}
interface IndexValidator {
@ -238,11 +248,12 @@ public class MetaDataCreateIndexService extends AbstractComponent {
private final AllocationService allocationService;
private final Settings settings;
private final IndexValidator validator;
private final IndexScopedSettings indexScopedSettings;
IndexCreationTask(Logger logger, AllocationService allocationService, CreateIndexClusterStateUpdateRequest request,
ActionListener<ClusterStateUpdateResponse> listener, IndicesService indicesService,
AliasValidator aliasValidator, NamedXContentRegistry xContentRegistry,
Settings settings, IndexValidator validator) {
Settings settings, IndexValidator validator, IndexScopedSettings indexScopedSettings) {
super(Priority.URGENT, request, listener);
this.request = request;
this.logger = logger;
@ -252,6 +263,7 @@ public class MetaDataCreateIndexService extends AbstractComponent {
this.xContentRegistry = xContentRegistry;
this.settings = settings;
this.validator = validator;
this.indexScopedSettings = indexScopedSettings;
}
@Override
@ -273,7 +285,8 @@ public class MetaDataCreateIndexService extends AbstractComponent {
// we only find a template when its an API call (a new index)
// find templates, highest order are better matching
List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(currentState.metaData(), request.index());
List<IndexTemplateMetaData> templates =
MetaDataIndexTemplateService.findTemplates(currentState.metaData(), request.index());
Map<String, Custom> customs = new HashMap<>();
@ -402,7 +415,14 @@ public class MetaDataCreateIndexService extends AbstractComponent {
if (recoverFromIndex != null) {
assert request.resizeType() != null;
prepareResizeIndexSettings(
currentState, mappings.keySet(), indexSettingsBuilder, recoverFromIndex, request.index(), request.resizeType());
currentState,
mappings.keySet(),
indexSettingsBuilder,
recoverFromIndex,
request.index(),
request.resizeType(),
request.copySettings(),
indexScopedSettings);
}
final Settings actualIndexSettings = indexSettingsBuilder.build();
tmpImdBuilder.settings(actualIndexSettings);
@ -673,8 +693,15 @@ public class MetaDataCreateIndexService extends AbstractComponent {
return sourceMetaData;
}
static void prepareResizeIndexSettings(ClusterState currentState, Set<String> mappingKeys, Settings.Builder indexSettingsBuilder,
Index resizeSourceIndex, String resizeIntoName, ResizeType type) {
static void prepareResizeIndexSettings(
final ClusterState currentState,
final Set<String> mappingKeys,
final Settings.Builder indexSettingsBuilder,
final Index resizeSourceIndex,
final String resizeIntoName,
final ResizeType type,
final boolean copySettings,
final IndexScopedSettings indexScopedSettings) {
final IndexMetaData sourceMetaData = currentState.metaData().index(resizeSourceIndex.getName());
if (type == ResizeType.SHRINK) {
final List<String> nodesToAllocateOn = validateShrinkIndex(currentState, resizeSourceIndex.getName(),
@ -695,15 +722,33 @@ public class MetaDataCreateIndexService extends AbstractComponent {
throw new IllegalStateException("unknown resize type is " + type);
}
final Predicate<String> sourceSettingsPredicate =
(s) -> (s.startsWith("index.similarity.") || s.startsWith("index.analysis.") || s.startsWith("index.sort."))
&& indexSettingsBuilder.keys().contains(s) == false;
final Settings.Builder builder = Settings.builder();
if (copySettings) {
// copy all settings and non-copyable settings and settings that have already been set (e.g., from the request)
for (final String key : sourceMetaData.getSettings().keySet()) {
final Setting<?> setting = indexScopedSettings.get(key);
if (setting == null) {
assert indexScopedSettings.isPrivateSetting(key) : key;
} else if (setting.getProperties().contains(Setting.Property.NotCopyableOnResize)) {
continue;
}
// do not override settings that have already been set (for example, from the request)
if (indexSettingsBuilder.keys().contains(key)) {
continue;
}
builder.copy(key, sourceMetaData.getSettings());
}
} else {
final Predicate<String> sourceSettingsPredicate =
(s) -> (s.startsWith("index.similarity.") || s.startsWith("index.analysis.") || s.startsWith("index.sort."))
&& indexSettingsBuilder.keys().contains(s) == false;
builder.put(sourceMetaData.getSettings().filter(sourceSettingsPredicate));
}
indexSettingsBuilder
// now copy all similarity / analysis / sort settings - this overrides all settings from the user unless they
// wanna add extra settings
.put(IndexMetaData.SETTING_VERSION_CREATED, sourceMetaData.getCreationVersion())
.put(IndexMetaData.SETTING_VERSION_UPGRADED, sourceMetaData.getUpgradedVersion())
.put(sourceMetaData.getSettings().filter(sourceSettingsPredicate))
.put(builder.build())
.put(IndexMetaData.SETTING_ROUTING_PARTITION_SIZE, sourceMetaData.getRoutingPartitionSize())
.put(IndexMetaData.INDEX_RESIZE_SOURCE_NAME.getKey(), resizeSourceIndex.getName())
.put(IndexMetaData.INDEX_RESIZE_SOURCE_UUID.getKey(), resizeSourceIndex.getUUID());

1
server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
View File

@ -43,6 +43,7 @@ import org.elasticsearch.indices.IndicesRequestCache;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;

17
server/src/main/java/org/elasticsearch/common/settings/Setting.java
View File

@ -114,7 +114,13 @@ public class Setting<T> implements ToXContentObject {
/**
* Index scope
*/
IndexScope
IndexScope,
/**
* Mark this setting as not copyable during an index resize (shrink or split). This property can only be applied to settings that
* also have {@link Property#IndexScope}.
*/
NotCopyableOnResize
}
private final Key key;
@ -142,10 +148,15 @@ public class Setting<T> implements ToXContentObject {
if (properties.length == 0) {
this.properties = EMPTY_PROPERTIES;
} else {
this.properties = EnumSet.copyOf(Arrays.asList(properties));
if (isDynamic() && isFinal()) {
final EnumSet<Property> propertiesAsSet = EnumSet.copyOf(Arrays.asList(properties));
if (propertiesAsSet.contains(Property.Dynamic) && propertiesAsSet.contains(Property.Final)) {
throw new IllegalArgumentException("final setting [" + key + "] cannot be dynamic");
}
if (propertiesAsSet.contains(Property.NotCopyableOnResize) && propertiesAsSet.contains(Property.IndexScope) == false) {
throw new IllegalArgumentException(
"non-index-scoped setting [" + key + "] can not have property [" + Property.NotCopyableOnResize + "]");
}
this.properties = propertiesAsSet;
}
}

14
server/src/main/java/org/elasticsearch/rest/action/admin/indices/RestResizeHandler.java
View File

@ -23,6 +23,7 @@ import org.elasticsearch.action.admin.indices.shrink.ResizeRequest;
import org.elasticsearch.action.admin.indices.shrink.ResizeType;
import org.elasticsearch.action.support.ActiveShardCount;
import org.elasticsearch.client.node.NodeClient;
import org.elasticsearch.common.Booleans;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.rest.BaseRestHandler;
import org.elasticsearch.rest.RestController;
@ -46,6 +47,19 @@ public abstract class RestResizeHandler extends BaseRestHandler {
public final RestChannelConsumer prepareRequest(final RestRequest request, final NodeClient client) throws IOException {
final ResizeRequest resizeRequest = new ResizeRequest(request.param("target"), request.param("index"));
resizeRequest.setResizeType(getResizeType());
final String rawCopySettings = request.param("copy_settings");
final boolean copySettings;
if (rawCopySettings == null) {
copySettings = resizeRequest.getCopySettings();
} else {
deprecationLogger.deprecated("parameter [copy_settings] is deprecated but was [" + rawCopySettings + "]");
if (rawCopySettings.length() == 0) {
copySettings = true;
} else {
copySettings = Booleans.parseBoolean(rawCopySettings);
}
}
resizeRequest.setCopySettings(copySettings);
request.applyContentParser(resizeRequest::fromXContent);
resizeRequest.timeout(request.paramAsTime("timeout", resizeRequest.timeout()));
resizeRequest.masterNodeTimeout(request.paramAsTime("master_timeout", resizeRequest.masterNodeTimeout()));

4
server/src/test/java/org/elasticsearch/cluster/metadata/IndexCreationTaskTests.java
View File

@ -39,6 +39,7 @@ import org.elasticsearch.cluster.routing.allocation.AllocationService;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.collect.ImmutableOpenMap;
import org.elasticsearch.common.compress.CompressedXContent;
import org.elasticsearch.common.settings.IndexScopedSettings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.util.set.Sets;
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
@ -388,8 +389,7 @@ public class IndexCreationTaskTests extends ESTestCase {
setupRequest();
final MetaDataCreateIndexService.IndexCreationTask task = new MetaDataCreateIndexService.IndexCreationTask(
logger, allocationService, request, listener, indicesService, aliasValidator, xContentRegistry, clusterStateSettings.build(),
validator
);
validator, IndexScopedSettings.DEFAULT_SCOPED_SETTINGS);
return task.execute(state);
}

222
server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataCreateIndexServiceTests.java
View File

@ -19,6 +19,7 @@
package org.elasticsearch.cluster.metadata;
import org.elasticsearch.ResourceAlreadyExistsException;
import org.elasticsearch.Version;
import org.elasticsearch.action.admin.indices.shrink.ResizeType;
import org.elasticsearch.cluster.ClusterName;
@ -34,22 +35,27 @@ import org.elasticsearch.cluster.routing.allocation.allocator.BalancedShardsAllo
import org.elasticsearch.cluster.routing.allocation.decider.AllocationDeciders;
import org.elasticsearch.cluster.routing.allocation.decider.MaxRetryAllocationDecider;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.IndexScopedSettings;
import org.elasticsearch.common.settings.Setting;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.IndexNotFoundException;
import org.elasticsearch.ResourceAlreadyExistsException;
import org.elasticsearch.indices.InvalidIndexNameException;
import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.test.VersionUtils;
import org.elasticsearch.test.gateway.TestGatewayAllocator;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.function.Consumer;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import static java.util.Collections.emptyMap;
import static java.util.Collections.min;
import static org.hamcrest.Matchers.endsWith;
import static org.hamcrest.Matchers.equalTo;
@ -228,90 +234,146 @@ public class MetaDataCreateIndexServiceTests extends ESTestCase {
Settings.builder().put("index.number_of_shards", targetShards).build());
}
public void testResizeIndexSettings() {
String indexName = randomAlphaOfLength(10);
List<Version> versions = Arrays.asList(VersionUtils.randomVersion(random()), VersionUtils.randomVersion(random()),
VersionUtils.randomVersion(random()));
public void testPrepareResizeIndexSettings() {
final List<Version> versions = Arrays.asList(VersionUtils.randomVersion(random()), VersionUtils.randomVersion(random()));
versions.sort(Comparator.comparingLong(l -> l.id));
Version version = versions.get(0);
Version minCompat = versions.get(1);
Version upgraded = versions.get(2);
// create one that won't fail
ClusterState clusterState = ClusterState.builder(createClusterState(indexName, randomIntBetween(2, 10), 0,
Settings.builder()
.put("index.blocks.write", true)
.put("index.similarity.default.type", "BM25")
.put("index.version.created", version)
.put("index.version.upgraded", upgraded)
.put("index.version.minimum_compatible", minCompat.luceneVersion.toString())
.put("index.analysis.analyzer.default.tokenizer", "keyword")
.build())).nodes(DiscoveryNodes.builder().add(newNode("node1")))
.build();
AllocationService service = new AllocationService(Settings.builder().build(), new AllocationDeciders(Settings.EMPTY,
Collections.singleton(new MaxRetryAllocationDecider(Settings.EMPTY))),
new TestGatewayAllocator(), new BalancedShardsAllocator(Settings.EMPTY), EmptyClusterInfoService.INSTANCE);
final Version version = versions.get(0);
final Version upgraded = versions.get(1);
final Settings indexSettings =
Settings.builder()
.put("index.version.created", version)
.put("index.version.upgraded", upgraded)
.put("index.similarity.default.type", "BM25")
.put("index.analysis.analyzer.default.tokenizer", "keyword")
.build();
runPrepareResizeIndexSettingsTest(
indexSettings,
Settings.EMPTY,
Collections.emptyList(),
randomBoolean(),
settings -> {
assertThat("similarity settings must be copied", settings.get("index.similarity.default.type"), equalTo("BM25"));
assertThat(
"analysis settings must be copied",
settings.get("index.analysis.analyzer.default.tokenizer"),
equalTo("keyword"));
assertThat(settings.get("index.routing.allocation.initial_recovery._id"), equalTo("node1"));
assertThat(settings.get("index.allocation.max_retries"), equalTo("1"));
assertThat(settings.getAsVersion("index.version.created", null), equalTo(version));
assertThat(settings.getAsVersion("index.version.upgraded", null), equalTo(upgraded));
});
}
RoutingTable routingTable = service.reroute(clusterState, "reroute").routingTable();
clusterState = ClusterState.builder(clusterState).routingTable(routingTable).build();
// now we start the shard
routingTable = service.applyStartedShards(clusterState,
routingTable.index(indexName).shardsWithState(ShardRoutingState.INITIALIZING)).routingTable();
clusterState = ClusterState.builder(clusterState).routingTable(routingTable).build();
{
final Settings.Builder builder = Settings.builder();
builder.put("index.number_of_shards", 1);
MetaDataCreateIndexService.prepareResizeIndexSettings(
clusterState,
Collections.emptySet(),
builder,
clusterState.metaData().index(indexName).getIndex(),
"target",
ResizeType.SHRINK);
final Settings settings = builder.build();
assertThat("similarity settings must be copied", settings.get("index.similarity.default.type"), equalTo("BM25"));
assertThat(
"analysis settings must be copied", settings.get("index.analysis.analyzer.default.tokenizer"), equalTo("keyword"));
assertThat(settings.get("index.routing.allocation.initial_recovery._id"), equalTo("node1"));
assertThat(settings.get("index.allocation.max_retries"), equalTo("1"));
assertThat(settings.getAsVersion("index.version.created", null), equalTo(version));
assertThat(settings.getAsVersion("index.version.upgraded", null), equalTo(upgraded));
}
public void testPrepareResizeIndexSettingsCopySettings() {
final int maxMergeCount = randomIntBetween(1, 16);
final int maxThreadCount = randomIntBetween(1, 16);
final Setting<String> nonCopyableExistingIndexSetting =
Setting.simpleString("index.non_copyable.existing", Setting.Property.IndexScope, Setting.Property.NotCopyableOnResize);
final Setting<String> nonCopyableRequestIndexSetting =
Setting.simpleString("index.non_copyable.request", Setting.Property.IndexScope, Setting.Property.NotCopyableOnResize);
runPrepareResizeIndexSettingsTest(
Settings.builder()
.put("index.merge.scheduler.max_merge_count", maxMergeCount)
.put("index.non_copyable.existing", "existing")
.build(),
Settings.builder()
.put("index.blocks.write", (String) null)
.put("index.merge.scheduler.max_thread_count", maxThreadCount)
.put("index.non_copyable.request", "request")
.build(),
Arrays.asList(nonCopyableExistingIndexSetting, nonCopyableRequestIndexSetting),
true,
settings -> {
assertNull(settings.getAsBoolean("index.blocks.write", null));
assertThat(settings.get("index.routing.allocation.require._name"), equalTo("node1"));
assertThat(settings.getAsInt("index.merge.scheduler.max_merge_count", null), equalTo(maxMergeCount));
assertThat(settings.getAsInt("index.merge.scheduler.max_thread_count", null), equalTo(maxThreadCount));
assertNull(settings.get("index.non_copyable.existing"));
assertThat(settings.get("index.non_copyable.request"), equalTo("request"));
});
}
public void testPrepareResizeIndexSettingsAnalysisSettings() {
// analysis settings from the request are not overwritten
{
final Settings.Builder builder = Settings.builder();
builder.put("index.number_of_shards", 1);
builder.put("index.analysis.analyzer.default.tokenizer", "whitespace");
MetaDataCreateIndexService.prepareResizeIndexSettings(
clusterState,
Collections.emptySet(),
builder,
clusterState.metaData().index(indexName).getIndex(),
"target",
ResizeType.SHRINK);
final Settings settings = builder.build();
assertThat(
"analysis settings are not overwritten",
settings.get("index.analysis.analyzer.default.tokenizer"),
equalTo("whitespace"));
}
runPrepareResizeIndexSettingsTest(
Settings.EMPTY,
Settings.builder().put("index.analysis.analyzer.default.tokenizer", "whitespace").build(),
Collections.emptyList(),
randomBoolean(),
settings ->
assertThat(
"analysis settings are not overwritten",
settings.get("index.analysis.analyzer.default.tokenizer"),
equalTo("whitespace"))
);
}
public void testPrepareResizeIndexSettingsSimilaritySettings() {
// similarity settings from the request are not overwritten
{
final Settings.Builder builder = Settings.builder();
builder.put("index.number_of_shards", 1);
builder.put("index.similarity.default.type", "DFR");
MetaDataCreateIndexService.prepareResizeIndexSettings(
clusterState,
Collections.emptySet(),
builder,
clusterState.metaData().index(indexName).getIndex(),
"target",
ResizeType.SHRINK);
final Settings settings = builder.build();
assertThat("similarity settings are not overwritten", settings.get("index.similarity.default.type"), equalTo("DFR"));
}
runPrepareResizeIndexSettingsTest(
Settings.EMPTY,
Settings.builder().put("index.similarity.sim.type", "DFR").build(),
Collections.emptyList(),
randomBoolean(),
settings ->
assertThat("similarity settings are not overwritten", settings.get("index.similarity.sim.type"), equalTo("DFR")));
}
private void runPrepareResizeIndexSettingsTest(
final Settings sourceSettings,
final Settings requestSettings,
final Collection<Setting<?>> additionalIndexScopedSettings,
final boolean copySettings,
final Consumer<Settings> consumer) {
final String indexName = randomAlphaOfLength(10);
final Settings indexSettings = Settings.builder()
.put("index.blocks.write", true)
.put("index.routing.allocation.require._name", "node1")
.put(sourceSettings)
.build();
final ClusterState initialClusterState =
ClusterState
.builder(createClusterState(indexName, randomIntBetween(2, 10), 0, indexSettings))
.nodes(DiscoveryNodes.builder().add(newNode("node1")))
.build();
final AllocationService service = new AllocationService(
Settings.builder().build(),
new AllocationDeciders(Settings.EMPTY,
Collections.singleton(new MaxRetryAllocationDecider(Settings.EMPTY))),
new TestGatewayAllocator(),
new BalancedShardsAllocator(Settings.EMPTY),
EmptyClusterInfoService.INSTANCE);
final RoutingTable initialRoutingTable = service.reroute(initialClusterState, "reroute").routingTable();
final ClusterState routingTableClusterState = ClusterState.builder(initialClusterState).routingTable(initialRoutingTable).build();
// now we start the shard
final RoutingTable routingTable = service.applyStartedShards(
routingTableClusterState,
initialRoutingTable.index(indexName).shardsWithState(ShardRoutingState.INITIALIZING)).routingTable();
final ClusterState clusterState = ClusterState.builder(routingTableClusterState).routingTable(routingTable).build();
final Settings.Builder indexSettingsBuilder = Settings.builder().put("index.number_of_shards", 1).put(requestSettings);
final Set<Setting<?>> settingsSet =
Stream.concat(
IndexScopedSettings.BUILT_IN_INDEX_SETTINGS.stream(),
additionalIndexScopedSettings.stream())
.collect(Collectors.toSet());
MetaDataCreateIndexService.prepareResizeIndexSettings(
clusterState,
Collections.emptySet(),
indexSettingsBuilder,
clusterState.metaData().index(indexName).getIndex(),
"target",
ResizeType.SHRINK,
copySettings,
new IndexScopedSettings(Settings.EMPTY, settingsSet));
consumer.accept(indexSettingsBuilder.build());
}
private DiscoveryNode newNode(String nodeId) {

9
server/src/test/java/org/elasticsearch/common/settings/SettingTests.java
View File

@ -722,12 +722,19 @@ public class SettingTests extends ESTestCase {
assertThat(ex.getMessage(), containsString("properties cannot be null for setting"));
}
public void testRejectConflictProperties() {
public void testRejectConflictingDynamicAndFinalProperties() {
IllegalArgumentException ex = expectThrows(IllegalArgumentException.class,
() -> Setting.simpleString("foo.bar", Property.Final, Property.Dynamic));
assertThat(ex.getMessage(), containsString("final setting [foo.bar] cannot be dynamic"));
}
public void testRejectNonIndexScopedNotCopyableOnResizeSetting() {
final IllegalArgumentException e = expectThrows(
IllegalArgumentException.class,
() -> Setting.simpleString("foo.bar", Property.NotCopyableOnResize));
assertThat(e, hasToString(containsString("non-index-scoped setting [foo.bar] can not have property [NotCopyableOnResize]")));
}
public void testTimeValue() {
final TimeValue random = TimeValue.parseTimeValue(randomTimeValue(), "test");

62
server/src/test/java/org/elasticsearch/rest/action/admin/indices/RestResizeHandlerTests.java Normal file
View File

@ -0,0 +1,62 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.rest.action.admin.indices;
import org.elasticsearch.client.node.NodeClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
import org.elasticsearch.rest.RestController;
import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.test.rest.FakeRestRequest;
import java.io.IOException;
import java.util.Collections;
import static org.mockito.Mockito.mock;
public class RestResizeHandlerTests extends ESTestCase {
public void testShrinkCopySettingsDeprecated() throws IOException {
final RestResizeHandler.RestShrinkIndexAction handler =
new RestResizeHandler.RestShrinkIndexAction(Settings.EMPTY, mock(RestController.class));
final String copySettings = randomFrom("true", "false");
final FakeRestRequest request =
new FakeRestRequest.Builder(NamedXContentRegistry.EMPTY)
.withParams(Collections.singletonMap("copy_settings", copySettings))
.withPath("source/_shrink/target")
.build();
handler.prepareRequest(request, mock(NodeClient.class));
assertWarnings("parameter [copy_settings] is deprecated but was [" + copySettings + "]");
}
public void testSplitCopySettingsDeprecated() throws IOException {
final RestResizeHandler.RestSplitIndexAction handler =
new RestResizeHandler.RestSplitIndexAction(Settings.EMPTY, mock(RestController.class));
final String copySettings = randomFrom("true", "false");
final FakeRestRequest request =
new FakeRestRequest.Builder(NamedXContentRegistry.EMPTY)
.withParams(Collections.singletonMap("copy_settings", copySettings))
.withPath("source/_split/target")
.build();
handler.prepareRequest(request, mock(NodeClient.class));
assertWarnings("parameter [copy_settings] is deprecated but was [" + copySettings + "]");
}
}