honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Introduce hidden indices (#51164) 

This change introduces a new feature for indices so that they can be
hidden from wildcard expansion. The feature is referred to as hidden
indices. An index can be marked hidden through the use of an index
setting, `index.hidden`, at creation time. One primary use case for
this feature is to have a construct that fits indices that are created
by the stack that contain data used for display to the user and/or
intended for querying by the user. The desire to keep them hidden is
to avoid confusing users when searching all of the data they have
indexed and getting results returned from indices created by the
system.

Hidden indices have the following properties:
* API calls for all indices (empty indices array, _all, or *) will not
  return hidden indices by default.
* Wildcard expansion will not return hidden indices by default unless
  the wildcard pattern begins with a `.`. This behavior is similar to
  shell expansion of wildcards.
* REST API calls can enable the expansion of wildcards to hidden
  indices with the `expand_wildcards` parameter. To expand wildcards
  to hidden indices, use the value `hidden` in conjunction with `open`
  and/or `closed`.
* Creation of a hidden index will ignore global index templates. A
  global index template is one with a match-all pattern.
* Index templates can make an index hidden, with the exception of a
  global index template.
* Accessing a hidden index directly requires no additional parameters.

Backport of #50452
This commit is contained in:
Jay Modi 2020-01-17 10:09:01 -07:00 committed by GitHub
parent 96e8f67425
commit 107989df3e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
27 changed files with 920 additions and 186 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/reference/index-modules.asciidoc
View File

@ -79,6 +79,13 @@ indices.
Indicates whether <<query-filter-context, cached filters>> are pre-loaded for
nested queries. Possible values are `true` (default) and `false`.
`index.hidden`::
Indicates whether the index should be hidden by default. Hidden indices are not
returned by default when using a wildcard expression. This behavior is controlled
per request through the use of the `expand_wildcards` parameter. Possible values are
`true` and `false` (default).
[float]
[[dynamic-index-settings]]
=== Dynamic index settings

49
docs/reference/rest-api/common-parms.asciidoc
View File

@ -63,7 +63,7 @@ Specifies what to do when the request:
--
* Contains wildcard expressions and there are no {transforms} that match.
* Contains the `_all` string or no identifiers and there are no matches.
* Contains wildcard expressions and there are only partial matches.
* Contains wildcard expressions and there are only partial matches.
The default value is `true`, which returns an empty `transforms` array when
there are no matches and the subset of results when there are partial matches.
@ -79,7 +79,7 @@ Specifies what to do when the request:
--
* Contains wildcard expressions and there are no {transforms} that match.
* Contains the `_all` string or no identifiers and there are no matches.
* Contains wildcard expressions and there are only partial matches.
* Contains wildcard expressions and there are only partial matches.
The default value is `true`, which returns a successful acknowledgement message
when there are no matches. When there are only partial matches, the API stops
@ -100,7 +100,7 @@ end::analyzer[]
tag::analyze_wildcard[]
`analyze_wildcard`::
(Optional, boolean) If `true`, wildcard and prefix queries are
(Optional, boolean) If `true`, wildcard and prefix queries are
analyzed. Defaults to `false`.
end::analyze_wildcard[]
@ -128,7 +128,7 @@ end::completion-fields[]
tag::default_operator[]
`default_operator`::
(Optional, string) The default operator for query string query: AND or OR.
(Optional, string) The default operator for query string query: AND or OR.
Defaults to `OR`.
end::default_operator[]
@ -154,7 +154,7 @@ end::detailed[]
tag::df[]
`df`::
(Optional, string) Field to use as default where no field prefix is
(Optional, string) Field to use as default where no field prefix is
given in the query string.
end::df[]
@ -191,6 +191,9 @@ Expand only to open indices.
`closed`::
Expand only to closed indices.
`hidden`::
Expansion of wildcards will include hidden indices.
`none`::
Wildcard expressions are not accepted.
--
@ -198,7 +201,7 @@ end::expand-wildcards[]
tag::field_statistics[]
`field_statistics`::
(Optional, boolean) If `true`, the response includes the document count, sum of document frequencies,
(Optional, boolean) If `true`, the response includes the document count, sum of document frequencies,
and sum of total term frequencies.
Defaults to `true`.
end::field_statistics[]
@ -324,7 +327,7 @@ end::if_seq_no[]
tag::ignore_throttled[]
`ignore_throttled`::
(Optional, boolean) If `true`, concrete, expanded or aliased indices are
(Optional, boolean) If `true`, concrete, expanded or aliased indices are
ignored when throttled.
end::ignore_throttled[]
@ -455,7 +458,7 @@ end::index-template[]
tag::lenient[]
`lenient`::
(Optional, boolean) If `true`, format-based query failures (such as
(Optional, boolean) If `true`, format-based query failures (such as
providing text to a numeric field) will be ignored. Defaults to `false`.
end::lenient[]
@ -626,13 +629,13 @@ end::search-q[]
tag::query[]
`query`::
(Optional, <<query-dsl,query object>>) Defines the search definition using the
(Optional, <<query-dsl,query object>>) Defines the search definition using the
<<query-dsl,Query DSL>>.
end::query[]
tag::realtime[]
`realtime`::
(Optional, boolean) If `true`, the request is real-time as opposed to near-real-time.
(Optional, boolean) If `true`, the request is real-time as opposed to near-real-time.
Defaults to `true`. See <<realtime>>.
end::realtime[]
@ -646,7 +649,7 @@ end::refresh[]
tag::request_cache[]
`request_cache`::
(Optional, boolean) If `true`, the request cache is used for this request.
(Optional, boolean) If `true`, the request cache is used for this request.
Defaults to the index-level setting.
end::request_cache[]
@ -676,14 +679,14 @@ end::cat-s[]
tag::scroll[]
`scroll`::
(Optional, <<time-units, time units>>) Specifies how long a consistent view of
(Optional, <<time-units, time units>>) Specifies how long a consistent view of
the index should be maintained for scrolled search.
end::scroll[]
tag::scroll_size[]
`scroll_size`::
(Optional, integer) Size of the scroll request that powers the operation.
Defaults to 100.
(Optional, integer) Size of the scroll request that powers the operation.
Defaults to 100.
end::scroll_size[]
tag::search_timeout[]
@ -739,7 +742,7 @@ end::size-transforms[]
tag::slices[]
`slices`::
(Optional, integer) The number of slices this task should be divided into.
(Optional, integer) The number of slices this task should be divided into.
Defaults to 1 meaning the task isn't sliced into subtasks.
end::slices[]
@ -750,24 +753,24 @@ end::sort[]
tag::source[]
`_source`::
(Optional, string) True or false to return the `_source` field or not, or a
(Optional, string) True or false to return the `_source` field or not, or a
list of fields to return.
end::source[]
tag::source_excludes[]
`_source_excludes`::
(Optional, string) A list of fields to exclude from the returned `_source`
(Optional, string) A list of fields to exclude from the returned `_source`
field.
end::source_excludes[]
tag::source_includes[]
`_source_includes`::
(Optional, string) A list of fields to extract and return from the `_source`
(Optional, string) A list of fields to extract and return from the `_source`
field.
end::source_includes[]
tag::source-transforms[]
The source of the data for the {transform}.
The source of the data for the {transform}.
end::source-transforms[]
tag::source-index-transforms[]
@ -831,13 +834,13 @@ end::task-id[]
tag::term_statistics[]
`term_statistics`::
(Optional, boolean) If `true`, the response includes term frequency and document frequency.
(Optional, boolean) If `true`, the response includes term frequency and document frequency.
Defaults to `false`.
end::term_statistics[]
tag::terminate_after[]
`terminate_after`::
(Optional, integer) The maximum number of documents to collect for each shard,
(Optional, integer) The maximum number of documents to collect for each shard,
upon reaching which the query execution will terminate early.
end::terminate_after[]
@ -886,7 +889,7 @@ end::transform-id-wildcard[]
tag::cat-v[]
`v`::
(Optional, boolean) If `true`, the response includes column headings.
(Optional, boolean) If `true`, the response includes column headings.
Defaults to `false`.
end::cat-v[]
@ -927,6 +930,6 @@ end::wait_for_active_shards[]
tag::wait_for_completion[]
`wait_for_completion`::
(Optional, boolean) If `true`, the request blocks until the operation is complete.
(Optional, boolean) If `true`, the request blocks until the operation is complete.
Defaults to `true`.
end::wait_for_completion[]

2
server/src/main/java/org/elasticsearch/action/admin/cluster/health/ClusterHealthRequest.java
View File

@ -39,7 +39,7 @@ import java.util.concurrent.TimeUnit;
public class ClusterHealthRequest extends MasterNodeReadRequest<ClusterHealthRequest> implements IndicesRequest.Replaceable {
private String[] indices;
private IndicesOptions indicesOptions = IndicesOptions.lenientExpand();
private IndicesOptions indicesOptions = IndicesOptions.lenientExpandHidden();
private TimeValue timeout = new TimeValue(30, TimeUnit.SECONDS);
private ClusterHealthStatus waitForStatus;
private boolean waitForNoRelocatingShards = false;

11
server/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java
View File

@ -43,7 +43,6 @@ import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
import org.elasticsearch.cluster.metadata.MetaData;
import org.elasticsearch.cluster.metadata.MetaDataCreateIndexService;
import org.elasticsearch.cluster.metadata.MetaDataIndexAliasesService;
import org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService;
import org.elasticsearch.cluster.service.ClusterService;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.inject.Inject;
@ -65,6 +64,7 @@ import java.util.regex.Pattern;
import java.util.stream.Collectors;
import static java.util.Collections.unmodifiableList;
import static org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.findTemplates;
/**
* Main class to swap the index pointed to by an alias, given some conditions
@ -131,7 +131,9 @@ public class TransportRolloverAction extends TransportMasterNodeAction<RolloverR
: generateRolloverIndexName(sourceProvidedName, indexNameExpressionResolver);
final String rolloverIndexName = indexNameExpressionResolver.resolveDateMathExpression(unresolvedName);
MetaDataCreateIndexService.validateIndexName(rolloverIndexName, state); // will fail if the index already exists
checkNoDuplicatedAliasInIndexTemplate(metaData, rolloverIndexName, rolloverRequest.getAlias());
final Boolean isHidden = IndexMetaData.INDEX_HIDDEN_SETTING.exists(rolloverRequest.getCreateIndexRequest().settings()) ?
IndexMetaData.INDEX_HIDDEN_SETTING.get(rolloverRequest.getCreateIndexRequest().settings()) : null;
checkNoDuplicatedAliasInIndexTemplate(metaData, rolloverIndexName, rolloverRequest.getAlias(), isHidden);
IndicesStatsRequest statsRequest = new IndicesStatsRequest().indices(rolloverRequest.getAlias())
.clear()
.indicesOptions(IndicesOptions.fromOptions(true, false, true, true))
@ -300,8 +302,9 @@ public class TransportRolloverAction extends TransportMasterNodeAction<RolloverR
* the rollover alias will point to multiple indices. This causes indexing requests to be rejected.
* To avoid this, we make sure that there is no duplicated alias in index templates before creating a new index.
*/
static void checkNoDuplicatedAliasInIndexTemplate(MetaData metaData, String rolloverIndexName, String rolloverRequestAlias) {
final List<IndexTemplateMetaData> matchedTemplates = MetaDataIndexTemplateService.findTemplates(metaData, rolloverIndexName);
static void checkNoDuplicatedAliasInIndexTemplate(MetaData metaData, String rolloverIndexName, String rolloverRequestAlias,
@Nullable Boolean isHidden) {
final List<IndexTemplateMetaData> matchedTemplates = findTemplates(metaData, rolloverIndexName, isHidden);
for (IndexTemplateMetaData template : matchedTemplates) {
if (template.aliases().containsKey(rolloverRequestAlias)) {
throw new IllegalArgumentException(String.format(Locale.ROOT,

2
server/src/main/java/org/elasticsearch/action/bulk/TransportBulkAction.java
View File

@ -311,7 +311,7 @@ public class TransportBulkAction extends HandledTransportAction<BulkRequest, Bul
}
} else if (indexRequest.index() != null) {
// the index does not exist yet (and this is a valid request), so match index templates to look for pipelines
List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(metaData, indexRequest.index());
List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(metaData, indexRequest.index(), null);
assert (templates != null);
// order of templates are highest order first
for (final IndexTemplateMetaData template : templates) {

11
server/src/main/java/org/elasticsearch/action/search/MultiSearchRequest.java
View File

@ -23,6 +23,7 @@ import org.elasticsearch.action.ActionRequest;
import org.elasticsearch.action.ActionRequestValidationException;
import org.elasticsearch.action.CompositeIndicesRequest;
import org.elasticsearch.action.support.IndicesOptions;
import org.elasticsearch.action.support.IndicesOptions.WildcardStates;
import org.elasticsearch.common.CheckedBiConsumer;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.io.stream.StreamInput;
@ -320,15 +321,7 @@ public class MultiSearchRequest extends ActionRequest implements CompositeIndice
xContentBuilder.field("index", request.indices());
}
if (request.indicesOptions() != null && request.indicesOptions() != SearchRequest.DEFAULT_INDICES_OPTIONS) {
if (request.indicesOptions().expandWildcardsOpen() && request.indicesOptions().expandWildcardsClosed()) {
xContentBuilder.field("expand_wildcards", "all");
} else if (request.indicesOptions().expandWildcardsOpen()) {
xContentBuilder.field("expand_wildcards", "open");
} else if (request.indicesOptions().expandWildcardsClosed()) {
xContentBuilder.field("expand_wildcards", "closed");
} else {
xContentBuilder.field("expand_wildcards", "none");
}
WildcardStates.toXContent(request.indicesOptions().getExpandWildcards(), xContentBuilder);
xContentBuilder.field("ignore_unavailable", request.indicesOptions().ignoreUnavailable());
xContentBuilder.field("allow_no_indices", request.indicesOptions().allowNoIndices());
}

137
server/src/main/java/org/elasticsearch/action/support/IndicesOptions.java
View File

@ -30,10 +30,9 @@ import org.elasticsearch.rest.RestRequest;
import java.io.IOException;
import java.util.Collection;
import java.util.EnumSet;
import java.util.HashSet;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
import static org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue;
import static org.elasticsearch.common.xcontent.support.XContentMapValues.nodeStringArrayValue;
@ -46,7 +45,8 @@ public class IndicesOptions implements ToXContentFragment {
public enum WildcardStates {
OPEN,
CLOSED;
CLOSED,
HIDDEN;
public static final EnumSet<WildcardStates> NONE = EnumSet.noneOf(WildcardStates.class);
@ -55,24 +55,45 @@ public class IndicesOptions implements ToXContentFragment {
return defaultStates;
}
Set<WildcardStates> states = new HashSet<>();
EnumSet<WildcardStates> states = EnumSet.noneOf(WildcardStates.class);
String[] wildcards = nodeStringArrayValue(value);
// TODO why do we let patterns like "none,all" or "open,none,closed" get used. The location of 'none' in the array changes the
// meaning of the resulting value
for (String wildcard : wildcards) {
if ("open".equals(wildcard)) {
states.add(OPEN);
} else if ("closed".equals(wildcard)) {
states.add(CLOSED);
} else if ("none".equals(wildcard)) {
states.clear();
} else if ("all".equals(wildcard)) {
states.add(OPEN);
states.add(CLOSED);
} else {
throw new IllegalArgumentException("No valid expand wildcard value [" + wildcard + "]");
switch (wildcard) {
case "open":
states.add(OPEN);
break;
case "closed":
states.add(CLOSED);
break;
case "hidden":
states.add(HIDDEN);
break;
case "none":
states.clear();
break;
case "all":
states = EnumSet.allOf(WildcardStates.class);
break;
default:
throw new IllegalArgumentException("No valid expand wildcard value [" + wildcard + "]");
}
}
return states.isEmpty() ? NONE : EnumSet.copyOf(states);
return states;
}
public static XContentBuilder toXContent(EnumSet<WildcardStates> states, XContentBuilder builder) throws IOException {
if (states.isEmpty()) {
builder.field("expand_wildcards", "none");
} else if (states.containsAll(EnumSet.allOf(WildcardStates.class))) {
builder.field("expand_wildcards", "all");
} else {
builder.field("expand_wildcards",
states.stream().map(state -> state.toString().toLowerCase(Locale.ROOT)).collect(Collectors.joining(",")));
}
return builder;
}
}
@ -92,9 +113,15 @@ public class IndicesOptions implements ToXContentFragment {
public static final IndicesOptions LENIENT_EXPAND_OPEN =
new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE),
EnumSet.of(WildcardStates.OPEN));
public static final IndicesOptions LENIENT_EXPAND_OPEN_HIDDEN =
new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE),
EnumSet.of(WildcardStates.OPEN, WildcardStates.HIDDEN));
public static final IndicesOptions LENIENT_EXPAND_OPEN_CLOSED =
new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE),
EnumSet.of(WildcardStates.OPEN, WildcardStates.CLOSED));
public static final IndicesOptions LENIENT_EXPAND_OPEN_CLOSED_HIDDEN =
new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE),
EnumSet.of(WildcardStates.OPEN, WildcardStates.CLOSED, WildcardStates.HIDDEN));
public static final IndicesOptions STRICT_EXPAND_OPEN_CLOSED =
new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES), EnumSet.of(WildcardStates.OPEN, WildcardStates.CLOSED));
public static final IndicesOptions STRICT_EXPAND_OPEN_FORBID_CLOSED =
@ -150,6 +177,13 @@ public class IndicesOptions implements ToXContentFragment {
return expandWildcards.contains(WildcardStates.CLOSED);
}
/**
* @return Whether wildcard expressions should get expanded to hidden indices
*/
public boolean expandWildcardsHidden() {
return expandWildcards.contains(WildcardStates.HIDDEN);
}
/**
* @return Whether execution on closed indices is allowed.
*/
@ -174,13 +208,19 @@ public class IndicesOptions implements ToXContentFragment {
}
/**
*
* @return whether indices that are marked as throttled should be ignored
*/
public boolean ignoreThrottled() {
return options.contains(Option.IGNORE_THROTTLED);
}
/**
* @return a copy of the {@link WildcardStates} that these indices options will expand to
*/
public EnumSet<WildcardStates> getExpandWildcards() {
return EnumSet.copyOf(expandWildcards);
}
public void writeIndicesOptions(StreamOutput out) throws IOException {
EnumSet<Option> options = this.options;
// never write this out to a pre 6.6 version
@ -189,30 +229,56 @@ public class IndicesOptions implements ToXContentFragment {
options.remove(Option.IGNORE_THROTTLED);
}
out.writeEnumSet(options);
out.writeEnumSet(expandWildcards);
if (out.getVersion().before(Version.V_7_7_0) && expandWildcards.contains(WildcardStates.HIDDEN)) {
final EnumSet<WildcardStates> states = EnumSet.copyOf(expandWildcards);
states.remove(WildcardStates.HIDDEN);
out.writeEnumSet(states);
} else {
out.writeEnumSet(expandWildcards);
}
}
public static IndicesOptions readIndicesOptions(StreamInput in) throws IOException {
return new IndicesOptions(in.readEnumSet(Option.class), in.readEnumSet(WildcardStates.class));
EnumSet<Option> options = in.readEnumSet(Option.class);
EnumSet<WildcardStates> states = in.readEnumSet(WildcardStates.class);
if (in.getVersion().before(Version.V_7_7_0)) {
states.add(WildcardStates.HIDDEN);
}
return new IndicesOptions(options, states);
}
public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
boolean expandToClosedIndices) {
return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices, true, false, false, false);
return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices, false);
}
public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
boolean expandToClosedIndices, boolean expandToHiddenIndices) {
return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices, expandToHiddenIndices, true,
false, false, false);
}
public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
boolean expandToClosedIndices, IndicesOptions defaultOptions) {
return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices,
defaultOptions.allowAliasesToMultipleIndices(), defaultOptions.forbidClosedIndices(), defaultOptions.ignoreAliases(),
defaultOptions.ignoreThrottled());
defaultOptions.expandWildcardsHidden(), defaultOptions.allowAliasesToMultipleIndices(),
defaultOptions.forbidClosedIndices(), defaultOptions.ignoreAliases(), defaultOptions.ignoreThrottled());
}
public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
boolean expandToClosedIndices, boolean allowAliasesToMultipleIndices, boolean forbidClosedIndices, boolean ignoreAliases,
boolean expandToClosedIndices, boolean allowAliasesToMultipleIndices,
boolean forbidClosedIndices, boolean ignoreAliases,
boolean ignoreThrottled) {
final Set<Option> opts = new HashSet<>();
final Set<WildcardStates> wildcards = new HashSet<>();
return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices, false,
allowAliasesToMultipleIndices, forbidClosedIndices, ignoreAliases, ignoreThrottled);
}
public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
boolean expandToClosedIndices, boolean expandToHiddenIndices,
boolean allowAliasesToMultipleIndices, boolean forbidClosedIndices, boolean ignoreAliases,
boolean ignoreThrottled) {
final EnumSet<Option> opts = EnumSet.noneOf(Option.class);
final EnumSet<WildcardStates> wildcards = EnumSet.noneOf(WildcardStates.class);
if (ignoreUnavailable) {
opts.add(Option.IGNORE_UNAVAILABLE);
@ -226,6 +292,9 @@ public class IndicesOptions implements ToXContentFragment {
if (expandToClosedIndices) {
wildcards.add(WildcardStates.CLOSED);
}
if (expandToHiddenIndices) {
wildcards.add(WildcardStates.HIDDEN);
}
if (allowAliasesToMultipleIndices == false) {
opts.add(Option.FORBID_ALIASES_TO_MULTIPLE_INDICES);
}
@ -285,6 +354,7 @@ public class IndicesOptions implements ToXContentFragment {
nodeBooleanValue(allowNoIndicesString, "allow_no_indices", defaultSettings.allowNoIndices()),
wildcards.contains(WildcardStates.OPEN),
wildcards.contains(WildcardStates.CLOSED),
wildcards.contains(WildcardStates.HIDDEN),
defaultSettings.allowAliasesToMultipleIndices(),
defaultSettings.forbidClosedIndices(),
defaultSettings.ignoreAliases(),
@ -355,6 +425,14 @@ public class IndicesOptions implements ToXContentFragment {
return LENIENT_EXPAND_OPEN;
}
/**
* @return indices options that ignores unavailable indices, expands wildcards to open and hidden indices, and
* allows that no indices are resolved from wildcard expressions (not returning an error).
*/
public static IndicesOptions lenientExpandOpenHidden() {
return LENIENT_EXPAND_OPEN_HIDDEN;
}
/**
* @return indices options that ignores unavailable indices, expands wildcards to both open and closed
* indices and allows that no indices are resolved from wildcard expressions (not returning an error).
@ -363,6 +441,14 @@ public class IndicesOptions implements ToXContentFragment {
return LENIENT_EXPAND_OPEN_CLOSED;
}
/**
* @return indices options that ignores unavailable indices, expands wildcards to all open and closed
* indices and allows that no indices are resolved from wildcard expressions (not returning an error).
*/
public static IndicesOptions lenientExpandHidden() {
return LENIENT_EXPAND_OPEN_CLOSED_HIDDEN;
}
@Override
public boolean equals(Object obj) {
if (obj == null) {
@ -390,6 +476,7 @@ public class IndicesOptions implements ToXContentFragment {
", allow_no_indices=" + allowNoIndices() +
", expand_wildcards_open=" + expandWildcardsOpen() +
", expand_wildcards_closed=" + expandWildcardsClosed() +
", expand_wildcards_hidden=" + expandWildcardsHidden() +
", allow_aliases_to_multiple_indices=" + allowAliasesToMultipleIndices() +
", forbid_closed_indices=" + forbidClosedIndices() +
", ignore_aliases=" + ignoreAliases() +

7
server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetaData.java
View File

@ -256,6 +256,13 @@ public class IndexMetaData implements Diffable<IndexMetaData>, ToXContentFragmen
Setting.Property.Dynamic,
Setting.Property.IndexScope);
/**
* Whether the index is considered hidden or not. A hidden index will not be resolved in
* normal wildcard searches unless explicitly allowed
*/
public static final Setting<Boolean> INDEX_HIDDEN_SETTING =
Setting.boolSetting("index.hidden", false, Property.IndexScope, Property.Final);
/**
* an internal index format description, allowing us to find out if this index is upgraded or needs upgrading
*/

32
server/src/main/java/org/elasticsearch/cluster/metadata/IndexNameExpressionResolver.java
View File

@ -162,7 +162,7 @@ public class IndexNameExpressionResolver {
for (ExpressionResolver expressionResolver : expressionResolvers) {
expressions = expressionResolver.resolve(context, expressions);
}
if (expressions.isEmpty()) {
if (!options.allowNoIndices()) {
IndexNotFoundException infe;
@ -191,7 +191,7 @@ public class IndexNameExpressionResolver {
if (expression.equals(MetaData.ALL)) {
infe = new IndexNotFoundException("no indices exist", expression);
} else {
infe = new IndexNotFoundException(expression);
infe = new IndexNotFoundException(expression);
}
infe.setResources("index_expression", expression);
throw infe;
@ -674,6 +674,8 @@ public class IndexNameExpressionResolver {
public List<String> resolve(Context context, List<String> expressions) {
IndicesOptions options = context.getOptions();
MetaData metaData = context.getState().metaData();
// only check open/closed since if we do not expand to open or closed it doesn't make sense to
// expand to hidden
if (options.expandWildcardsClosed() == false && options.expandWildcardsOpen() == false) {
return expressions;
}
@ -743,7 +745,7 @@ public class IndexNameExpressionResolver {
final IndexMetaData.State excludeState = excludeState(options);
final Map<String, AliasOrIndex> matches = matches(context, metaData, expression);
Set<String> expand = expand(context, excludeState, matches);
Set<String> expand = expand(context, excludeState, matches, expression, options.expandWildcardsHidden());
if (add) {
result.addAll(expand);
} else {
@ -838,7 +840,8 @@ public class IndexNameExpressionResolver {
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
}
private static Set<String> expand(Context context, IndexMetaData.State excludeState, Map<String, AliasOrIndex> matches) {
private static Set<String> expand(Context context, IndexMetaData.State excludeState, Map<String, AliasOrIndex> matches,
String expression, boolean includeHidden) {
Set<String> expand = new HashSet<>();
for (Map.Entry<String, AliasOrIndex> entry : matches.entrySet()) {
AliasOrIndex aliasOrIndex = entry.getValue();
@ -847,7 +850,14 @@ public class IndexNameExpressionResolver {
} else {
for (IndexMetaData meta : aliasOrIndex.getIndices()) {
if (excludeState == null || meta.getState() != excludeState) {
expand.add(meta.getIndex().getName());
if (includeHidden) {
expand.add(meta.getIndex().getName());
} else if (IndexMetaData.INDEX_HIDDEN_SETTING.get(meta.getSettings()) == false) {
expand.add(meta.getIndex().getName());
} else if (meta.getIndex().getName().startsWith(".") &&
expression.startsWith(".") && Regex.isSimpleMatchPattern(expression)) {
expand.add(meta.getIndex().getName());
}
}
}
}
@ -861,12 +871,18 @@ public class IndexNameExpressionResolver {
}
private static List<String> resolveEmptyOrTrivialWildcard(IndicesOptions options, MetaData metaData) {
if (options.expandWildcardsOpen() && options.expandWildcardsClosed()) {
if (options.expandWildcardsOpen() && options.expandWildcardsClosed() && options.expandWildcardsHidden()) {
return Arrays.asList(metaData.getConcreteAllIndices());
} else if (options.expandWildcardsOpen()) {
} else if (options.expandWildcardsOpen() && options.expandWildcardsClosed()) {
return Arrays.asList(metaData.getConcreteVisibleIndices());
} else if (options.expandWildcardsOpen() && options.expandWildcardsHidden()) {
return Arrays.asList(metaData.getConcreteAllOpenIndices());
} else if (options.expandWildcardsClosed()) {
} else if (options.expandWildcardsOpen()) {
return Arrays.asList(metaData.getConcreteVisibleOpenIndices());
} else if (options.expandWildcardsClosed() && options.expandWildcardsHidden()) {
return Arrays.asList(metaData.getConcreteAllClosedIndices());
} else if (options.expandWildcardsClosed()) {
return Arrays.asList(metaData.getConcreteVisibleClosedIndices());
} else {
return Collections.emptyList();
}

66
server/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java
View File

@ -179,15 +179,19 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
private final int totalOpenIndexShards;
private final String[] allIndices;
private final String[] visibleIndices;
private final String[] allOpenIndices;
private final String[] visibleOpenIndices;
private final String[] allClosedIndices;
private final String[] visibleClosedIndices;
private final SortedMap<String, AliasOrIndex> aliasAndIndexLookup;
MetaData(String clusterUUID, boolean clusterUUIDCommitted, long version, CoordinationMetaData coordinationMetaData,
Settings transientSettings, Settings persistentSettings, DiffableStringMap hashesOfConsistentSettings,
ImmutableOpenMap<String, IndexMetaData> indices, ImmutableOpenMap<String, IndexTemplateMetaData> templates,
ImmutableOpenMap<String, Custom> customs, String[] allIndices, String[] allOpenIndices, String[] allClosedIndices,
ImmutableOpenMap<String, Custom> customs, String[] allIndices, String[] visibleIndices, String[] allOpenIndices,
String[] visibleOpenIndices, String[] allClosedIndices, String[] visibleClosedIndices,
SortedMap<String, AliasOrIndex> aliasAndIndexLookup) {
this.clusterUUID = clusterUUID;
this.clusterUUIDCommitted = clusterUUIDCommitted;
@ -212,8 +216,11 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
this.totalOpenIndexShards = totalOpenIndexShards;
this.allIndices = allIndices;
this.visibleIndices = visibleIndices;
this.allOpenIndices = allOpenIndices;
this.visibleOpenIndices = visibleOpenIndices;
this.allClosedIndices = allClosedIndices;
this.visibleClosedIndices = visibleClosedIndices;
this.aliasAndIndexLookup = aliasAndIndexLookup;
}
@ -538,14 +545,41 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
return allIndices;
}
/**
* Returns all the concrete indices that are not hidden.
*/
public String[] getConcreteVisibleIndices() {
return visibleIndices;
}
/**
* Returns all of the concrete indices that are open.
*/
public String[] getConcreteAllOpenIndices() {
return allOpenIndices;
}
/**
* Returns all of the concrete indices that are open and not hidden.
*/
public String[] getConcreteVisibleOpenIndices() {
return visibleOpenIndices;
}
/**
* Returns all of the concrete indices that are closed.
*/
public String[] getConcreteAllClosedIndices() {
return allClosedIndices;
}
/**
* Returns all of the concrete indices that are closed and not hidden.
*/
public String[] getConcreteVisibleClosedIndices() {
return visibleClosedIndices;
}
/**
* Returns indexing routing for the given <code>aliasOrIndex</code>. Resolves routing from the alias metadata used
* in the write index.
@ -1223,18 +1257,31 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
// 2) The aliasAndIndexLookup can be updated instead of rebuilding it all the time.
final Set<String> allIndices = new HashSet<>(indices.size());
final List<String> visibleIndices = new ArrayList<>();
final List<String> allOpenIndices = new ArrayList<>();
final List<String> visibleOpenIndices = new ArrayList<>();
final List<String> allClosedIndices = new ArrayList<>();
final List<String> visibleClosedIndices = new ArrayList<>();
final Set<String> duplicateAliasesIndices = new HashSet<>();
for (ObjectCursor<IndexMetaData> cursor : indices.values()) {
final IndexMetaData indexMetaData = cursor.value;
final String name = indexMetaData.getIndex().getName();
boolean added = allIndices.add(name);
assert added : "double index named [" + name + "]";
final boolean visible = IndexMetaData.INDEX_HIDDEN_SETTING.get(indexMetaData.getSettings()) == false;
if (visible) {
visibleIndices.add(name);
}
if (indexMetaData.getState() == IndexMetaData.State.OPEN) {
allOpenIndices.add(indexMetaData.getIndex().getName());
allOpenIndices.add(name);
if (visible) {
visibleOpenIndices.add(name);
}
} else if (indexMetaData.getState() == IndexMetaData.State.CLOSE) {
allClosedIndices.add(indexMetaData.getIndex().getName());
allClosedIndices.add(name);
if (visible) {
visibleClosedIndices.add(name);
}
}
indexMetaData.getAliases().keysIt().forEachRemaining(duplicateAliasesIndices::add);
}
@ -1262,13 +1309,16 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
// TODO: I think we can remove these arrays. it isn't worth the effort, for operations on all indices.
// When doing an operation across all indices, most of the time is spent on actually going to all shards and
// do the required operations, the bottleneck isn't resolving expressions into concrete indices.
String[] allIndicesArray = allIndices.toArray(new String[allIndices.size()]);
String[] allOpenIndicesArray = allOpenIndices.toArray(new String[allOpenIndices.size()]);
String[] allClosedIndicesArray = allClosedIndices.toArray(new String[allClosedIndices.size()]);
String[] allIndicesArray = allIndices.toArray(Strings.EMPTY_ARRAY);
String[] visibleIndicesArray = visibleIndices.toArray(Strings.EMPTY_ARRAY);
String[] allOpenIndicesArray = allOpenIndices.toArray(Strings.EMPTY_ARRAY);
String[] visibleOpenIndicesArray = visibleOpenIndices.toArray(Strings.EMPTY_ARRAY);
String[] allClosedIndicesArray = allClosedIndices.toArray(Strings.EMPTY_ARRAY);
String[] visibleClosedIndicesArray = visibleClosedIndices.toArray(Strings.EMPTY_ARRAY);
return new MetaData(clusterUUID, clusterUUIDCommitted, version, coordinationMetaData, transientSettings, persistentSettings,
hashesOfConsistentSettings, indices.build(), templates.build(), customs.build(), allIndicesArray, allOpenIndicesArray,
allClosedIndicesArray, aliasAndIndexLookup);
hashesOfConsistentSettings, indices.build(), templates.build(), customs.build(), allIndicesArray, visibleIndicesArray,
allOpenIndicesArray, visibleOpenIndicesArray, allClosedIndicesArray, visibleClosedIndicesArray, aliasAndIndexLookup);
}
private SortedMap<String, AliasOrIndex> buildAliasAndIndexLookup() {

4
server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataCreateIndexService.java
View File

@ -274,8 +274,10 @@ public class MetaDataCreateIndexService {
// we only find a template when its an API call (a new index)
// find templates, highest order are better matching
final Boolean isHidden = IndexMetaData.INDEX_HIDDEN_SETTING.exists(request.settings()) ?
IndexMetaData.INDEX_HIDDEN_SETTING.get(request.settings()) : null;
final List<IndexTemplateMetaData> templates = sourceMetaData == null ?
Collections.unmodifiableList(MetaDataIndexTemplateService.findTemplates(currentState.metaData(), request.index())) :
Collections.unmodifiableList(MetaDataIndexTemplateService.findTemplates(currentState.metaData(), request.index(), isHidden)) :
Collections.emptyList();
final Map<String, Map<String, Object>> mappings = Collections.unmodifiableMap(parseMappings(request.mappings(), templates,

64
server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java
View File

@ -28,6 +28,7 @@ import org.elasticsearch.action.support.master.MasterNodeRequest;
import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.ClusterStateUpdateTask;
import org.elasticsearch.cluster.service.ClusterService;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.Priority;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.UUIDs;
@ -55,7 +56,9 @@ import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.function.Predicate;
import static org.elasticsearch.indices.cluster.IndicesClusterStateService.AllocatedIndices.IndexRemovalReason.NO_LONGER_ASSIGNED;
@ -196,19 +199,61 @@ public class MetaDataIndexTemplateService {
}
/**
* Finds index templates whose index pattern matched with the given index name.
* The result is sorted by {@link IndexTemplateMetaData#order} descending.
* Finds index templates whose index pattern matched with the given index name. In the case of
* hidden indices, a template with a match all pattern or global template will not be returned.
*
* @param metaData The {@link MetaData} containing all of the {@link IndexTemplateMetaData} values
* @param indexName The name of the index that templates are being found for
* @param isHidden Whether or not the index is known to be hidden. May be {@code null} if the index
* being hidden has not been explicitly requested. When {@code null} if the result
* of template application results in a hidden index, then global templates will
* not be returned
* @return a list of templates sorted by {@link IndexTemplateMetaData#order()} descending.
*
*/
public static List<IndexTemplateMetaData> findTemplates(MetaData metaData, String indexName) {
public static List<IndexTemplateMetaData> findTemplates(MetaData metaData, String indexName, @Nullable Boolean isHidden) {
final Predicate<String> patternMatchPredicate = pattern -> Regex.simpleMatch(pattern, indexName);
final List<IndexTemplateMetaData> matchedTemplates = new ArrayList<>();
for (ObjectCursor<IndexTemplateMetaData> cursor : metaData.templates().values()) {
final IndexTemplateMetaData template = cursor.value;
final boolean matched = template.patterns().stream().anyMatch(pattern -> Regex.simpleMatch(pattern, indexName));
if (matched) {
matchedTemplates.add(template);
if (isHidden == null || isHidden == Boolean.FALSE) {
final boolean matched = template.patterns().stream().anyMatch(patternMatchPredicate);
if (matched) {
matchedTemplates.add(template);
}
} else {
assert isHidden == Boolean.TRUE;
final boolean isNotMatchAllTemplate = template.patterns().stream().noneMatch(Regex::isMatchAllPattern);
if (isNotMatchAllTemplate) {
if (template.patterns().stream().anyMatch(patternMatchPredicate)) {
matchedTemplates.add(template);
}
}
}
}
CollectionUtil.timSort(matchedTemplates, Comparator.comparingInt(IndexTemplateMetaData::order).reversed());
// this is complex but if the index is not hidden in the create request but is hidden as the result of template application,
// then we need to exclude global templates
if (isHidden == null) {
final Optional<IndexTemplateMetaData> templateWithHiddenSetting = matchedTemplates.stream()
.filter(template -> IndexMetaData.INDEX_HIDDEN_SETTING.exists(template.settings())).findFirst();
if (templateWithHiddenSetting.isPresent()) {
final boolean templatedIsHidden = IndexMetaData.INDEX_HIDDEN_SETTING.get(templateWithHiddenSetting.get().settings());
if (templatedIsHidden) {
// remove the global templates
matchedTemplates.removeIf(current -> current.patterns().stream().anyMatch(Regex::isMatchAllPattern));
}
// validate that hidden didn't change
final Optional<IndexTemplateMetaData> templateWithHiddenSettingPostRemoval = matchedTemplates.stream()
.filter(template -> IndexMetaData.INDEX_HIDDEN_SETTING.exists(template.settings())).findFirst();
if (templateWithHiddenSettingPostRemoval.isPresent() == false ||
templateWithHiddenSetting.get() != templateWithHiddenSettingPostRemoval.get()) {
throw new IllegalStateException("A global index template [" + templateWithHiddenSetting.get().name() +
"] defined the index hidden setting, which is not allowed");
}
}
}
return matchedTemplates;
}
@ -304,6 +349,13 @@ public class MetaDataIndexTemplateService {
}
List<String> indexSettingsValidation = metaDataCreateIndexService.getIndexSettingsValidationErrors(request.settings, true);
validationErrors.addAll(indexSettingsValidation);
if (request.indexPatterns.stream().anyMatch(Regex::isMatchAllPattern)) {
if (IndexMetaData.INDEX_HIDDEN_SETTING.exists(request.settings)) {
validationErrors.add("global templates may not specify the setting " + IndexMetaData.INDEX_HIDDEN_SETTING.getKey());
}
}
if (!validationErrors.isEmpty()) {
ValidationException validationException = new ValidationException();
validationException.addValidationErrors(validationErrors);

1
server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
View File

@ -80,6 +80,7 @@ public final class IndexScopedSettings extends AbstractScopedSettings {
IndexMetaData.INDEX_PRIORITY_SETTING,
IndexMetaData.INDEX_DATA_PATH_SETTING,
IndexMetaData.INDEX_FORMAT_SETTING,
IndexMetaData.INDEX_HIDDEN_SETTING,
SearchSlowLog.INDEX_SEARCH_SLOWLOG_THRESHOLD_FETCH_DEBUG_SETTING,
SearchSlowLog.INDEX_SEARCH_SLOWLOG_THRESHOLD_FETCH_WARN_SETTING,
SearchSlowLog.INDEX_SEARCH_SLOWLOG_THRESHOLD_FETCH_INFO_SETTING,

14
server/src/test/java/org/elasticsearch/action/OriginalIndicesTests.java
View File

@ -19,6 +19,7 @@
package org.elasticsearch.action;
import org.elasticsearch.Version;
import org.elasticsearch.action.support.IndicesOptions;
import org.elasticsearch.common.io.stream.BytesStreamOutput;
import org.elasticsearch.common.io.stream.StreamInput;
@ -26,7 +27,7 @@ import org.elasticsearch.test.ESTestCase;
import java.io.IOException;
import static org.elasticsearch.test.VersionUtils.randomVersion;
import static org.elasticsearch.test.VersionUtils.randomCompatibleVersion;
import static org.hamcrest.CoreMatchers.equalTo;
public class OriginalIndicesTests extends ESTestCase {
@ -41,7 +42,7 @@ public class OriginalIndicesTests extends ESTestCase {
OriginalIndices originalIndices = randomOriginalIndices();
BytesStreamOutput out = new BytesStreamOutput();
out.setVersion(randomVersion(random()));
out.setVersion(randomCompatibleVersion(random(), Version.CURRENT));
OriginalIndices.writeOriginalIndices(originalIndices, out);
StreamInput in = out.bytes().streamInput();
@ -49,7 +50,14 @@ public class OriginalIndicesTests extends ESTestCase {
OriginalIndices originalIndices2 = OriginalIndices.readOriginalIndices(in);
assertThat(originalIndices2.indices(), equalTo(originalIndices.indices()));
assertThat(originalIndices2.indicesOptions(), equalTo(originalIndices.indicesOptions()));
// indices options are not equivalent when sent to an older version and re-read due
// to the addition of hidden indices as expand to hidden indices is always true when
// read from a prior version
if (out.getVersion().onOrAfter(Version.V_7_7_0) || originalIndices.indicesOptions().expandWildcardsHidden()) {
assertThat(originalIndices2.indicesOptions(), equalTo(originalIndices.indicesOptions()));
} else if (originalIndices.indicesOptions().expandWildcardsHidden()) {
assertThat(originalIndices2.indicesOptions(), equalTo(originalIndices.indicesOptions()));
}
}
}

5
server/src/test/java/org/elasticsearch/action/admin/cluster/health/ClusterHealthRequestTests.java
View File

@ -55,6 +55,11 @@ public class ClusterHealthRequestTests extends ESTestCase {
assertThat(cloneRequest.indicesOptions(), equalTo(originalRequest.indicesOptions()));
}
public void testRequestReturnsHiddenIndicesByDefault() {
final ClusterHealthRequest defaultRequest = new ClusterHealthRequest();
assertTrue(defaultRequest.indicesOptions().expandWildcardsHidden());
}
public void testBwcSerialization() throws Exception {
for (int runs = 0; runs < randomIntBetween(5, 20); runs++) {
// Generate a random cluster health request in version < 7.2.0 and serializes it

11
server/src/test/java/org/elasticsearch/action/admin/cluster/shards/ClusterSearchShardsRequestTests.java
View File

@ -40,7 +40,7 @@ public class ClusterSearchShardsRequestTests extends ESTestCase {
}
if (randomBoolean()) {
request.indicesOptions(
IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean()));
IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean()));
}
if (randomBoolean()) {
request.preference(randomAlphaOfLengthBetween(3, 10));
@ -54,7 +54,7 @@ public class ClusterSearchShardsRequestTests extends ESTestCase {
request.routing(routings);
}
Version version = VersionUtils.randomVersionBetween(random(), Version.V_6_0_0, Version.CURRENT);
Version version = VersionUtils.randomCompatibleVersion(random(), Version.CURRENT);
try (BytesStreamOutput out = new BytesStreamOutput()) {
out.setVersion(version);
request.writeTo(out);
@ -62,7 +62,12 @@ public class ClusterSearchShardsRequestTests extends ESTestCase {
in.setVersion(version);
ClusterSearchShardsRequest deserialized = new ClusterSearchShardsRequest(in);
assertArrayEquals(request.indices(), deserialized.indices());
assertEquals(request.indicesOptions(), deserialized.indicesOptions());
// indices options are not equivalent when sent to an older version and re-read due
// to the addition of hidden indices as expand to hidden indices is always true when
// read from a prior version
if (version.onOrAfter(Version.V_7_7_0) || request.indicesOptions().expandWildcardsHidden()) {
assertEquals(request.indicesOptions(), deserialized.indicesOptions());
}
assertEquals(request.routing(), deserialized.routing());
assertEquals(request.preference(), deserialized.preference());
}

41
server/src/test/java/org/elasticsearch/action/admin/indices/close/CloseIndexRequestTests.java
View File

@ -28,8 +28,6 @@ import org.elasticsearch.tasks.TaskId;
import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.test.VersionUtils;
import static org.elasticsearch.test.VersionUtils.randomVersionBetween;
public class CloseIndexRequestTests extends ESTestCase {
public void testSerialization() throws Exception {
@ -54,38 +52,64 @@ public class CloseIndexRequestTests extends ESTestCase {
{
final CloseIndexRequest request = randomRequest();
try (BytesStreamOutput out = new BytesStreamOutput()) {
out.setVersion(randomVersionBetween(random(), Version.V_6_4_0, VersionUtils.getPreviousVersion(Version.V_7_2_0)));
out.setVersion(VersionUtils.randomCompatibleVersion(random(), Version.CURRENT));
request.writeTo(out);
try (StreamInput in = out.bytes().streamInput()) {
in.setVersion(out.getVersion());
assertEquals(request.getParentTask(), TaskId.readFromStream(in));
assertEquals(request.masterNodeTimeout(), in.readTimeValue());
assertEquals(request.timeout(), in.readTimeValue());
assertArrayEquals(request.indices(), in.readStringArray());
assertEquals(request.indicesOptions(), IndicesOptions.readIndicesOptions(in));
// indices options are not equivalent when sent to an older version and re-read due
// to the addition of hidden indices as expand to hidden indices is always true when
// read from a prior version
final IndicesOptions indicesOptions = IndicesOptions.readIndicesOptions(in);
if (out.getVersion().onOrAfter(Version.V_7_7_0) || request.indicesOptions().expandWildcardsHidden()) {
assertEquals(request.indicesOptions(), indicesOptions);
}
if (in.getVersion().onOrAfter(Version.V_7_2_0)) {
assertEquals(request.waitForActiveShards(), ActiveShardCount.readFrom(in));
} else {
assertEquals(0, in.available());
}
}
}
}
{
final CloseIndexRequest sample = randomRequest();
final Version version = VersionUtils.randomCompatibleVersion(random(), Version.CURRENT);
try (BytesStreamOutput out = new BytesStreamOutput()) {
out.setVersion(version);
sample.getParentTask().writeTo(out);
out.writeTimeValue(sample.masterNodeTimeout());
out.writeTimeValue(sample.timeout());
out.writeStringArray(sample.indices());
sample.indicesOptions().writeIndicesOptions(out);
if (out.getVersion().onOrAfter(Version.V_7_2_0)) {
sample.waitForActiveShards().writeTo(out);
}
final CloseIndexRequest deserializedRequest;
try (StreamInput in = out.bytes().streamInput()) {
in.setVersion(randomVersionBetween(random(), Version.V_6_4_0, VersionUtils.getPreviousVersion(Version.V_7_2_0)));
in.setVersion(version);
deserializedRequest = new CloseIndexRequest(in);
}
assertEquals(sample.getParentTask(), deserializedRequest.getParentTask());
assertEquals(sample.masterNodeTimeout(), deserializedRequest.masterNodeTimeout());
assertEquals(sample.timeout(), deserializedRequest.timeout());
assertArrayEquals(sample.indices(), deserializedRequest.indices());
assertEquals(sample.indicesOptions(), deserializedRequest.indicesOptions());
assertEquals(ActiveShardCount.NONE, deserializedRequest.waitForActiveShards());
// indices options are not equivalent when sent to an older version and re-read due
// to the addition of hidden indices as expand to hidden indices is always true when
// read from a prior version
if (out.getVersion().onOrAfter(Version.V_7_7_0) || sample.indicesOptions().expandWildcardsHidden()) {
assertEquals(sample.indicesOptions(), deserializedRequest.indicesOptions());
}
if (out.getVersion().onOrAfter(Version.V_7_2_0)) {
assertEquals(sample.waitForActiveShards(), deserializedRequest.waitForActiveShards());
} else {
assertEquals(ActiveShardCount.NONE, deserializedRequest.waitForActiveShards());
}
}
}
}
@ -94,7 +118,8 @@ public class CloseIndexRequestTests extends ESTestCase {
CloseIndexRequest request = new CloseIndexRequest();
request.indices(generateRandomStringArray(10, 5, false, false));
if (randomBoolean()) {
request.indicesOptions(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean()));
request.indicesOptions(
IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean()));
}
if (randomBoolean()) {
request.timeout(randomPositiveTimeValue());

20
server/src/test/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverActionTests.java
View File

@ -79,6 +79,7 @@ import org.mockito.ArgumentCaptor;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
@ -359,7 +360,24 @@ public class TransportRolloverActionTests extends ESTestCase {
String indexName = randomFrom("foo-123", "bar-xyz");
String aliasName = randomFrom("foo-write", "bar-write");
final IllegalArgumentException ex = expectThrows(IllegalArgumentException.class,
() -> TransportRolloverAction.checkNoDuplicatedAliasInIndexTemplate(metaData, indexName, aliasName));
() -> TransportRolloverAction.checkNoDuplicatedAliasInIndexTemplate(metaData, indexName, aliasName, randomBoolean()));
assertThat(ex.getMessage(), containsString("index template [test-template]"));
}
public void testHiddenAffectsResolvedTemplates() {
final IndexTemplateMetaData template = IndexTemplateMetaData.builder("test-template")
.patterns(Collections.singletonList("*"))
.putAlias(AliasMetaData.builder("foo-write")).putAlias(AliasMetaData.builder("bar-write").writeIndex(randomBoolean()))
.build();
final MetaData metaData = MetaData.builder().put(createMetaData(randomAlphaOfLengthBetween(5, 7)), false).put(template).build();
String indexName = randomFrom("foo-123", "bar-xyz");
String aliasName = randomFrom("foo-write", "bar-write");
// hidden shouldn't throw
TransportRolloverAction.checkNoDuplicatedAliasInIndexTemplate(metaData, indexName, aliasName, Boolean.TRUE);
// not hidden will throw
final IllegalArgumentException ex = expectThrows(IllegalArgumentException.class, () ->
TransportRolloverAction.checkNoDuplicatedAliasInIndexTemplate(metaData, indexName, aliasName, randomFrom(Boolean.FALSE, null)));
assertThat(ex.getMessage(), containsString("index template [test-template]"));
}

75
server/src/test/java/org/elasticsearch/action/admin/indices/template/put/MetaDataIndexTemplateServiceTests.java
View File

@ -48,17 +48,20 @@ import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.stream.Collectors;
import static java.util.Collections.singletonList;
import static org.elasticsearch.common.settings.Settings.builder;
import static org.hamcrest.CoreMatchers.containsString;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.CoreMatchers.instanceOf;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.empty;
import static org.hamcrest.Matchers.is;
public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
public void testIndexTemplateInvalidNumberOfShards() {
PutRequest request = new PutRequest("test", "test_shards");
request.patterns(Collections.singletonList("test_shards*"));
request.patterns(singletonList("test_shards*"));
request.settings(builder()
.put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, "0")
@ -76,7 +79,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
public void testIndexTemplateValidationAccumulatesValidationErrors() {
PutRequest request = new PutRequest("test", "putTemplate shards");
request.patterns(Collections.singletonList("_test_shards*"));
request.patterns(singletonList("_test_shards*"));
request.settings(builder().put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, "0").build());
List<Throwable> throwables = putTemplate(xContentRegistry(), request);
@ -101,7 +104,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
public void testIndexTemplateWithValidateMapping() throws Exception {
PutRequest request = new PutRequest("api", "validate_template");
request.patterns(Collections.singletonList("te*"));
request.patterns(singletonList("te*"));
request.putMapping("type1", Strings.toString(XContentFactory.jsonBuilder().startObject().startObject("type1")
.startObject("properties").startObject("field2").field("type", "text").field("analyzer", "custom_1").endObject()
.endObject().endObject().endObject()));
@ -114,7 +117,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
public void testBrokenMapping() throws Exception {
PutRequest request = new PutRequest("api", "broken_mapping");
request.patterns(Collections.singletonList("te*"));
request.patterns(singletonList("te*"));
request.putMapping("type1", "abcde");
List<Throwable> errors = putTemplateDetail(request);
@ -126,7 +129,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
public void testAliasInvalidFilterInvalidJson() throws Exception {
//invalid json: put index template fails
PutRequest request = new PutRequest("api", "blank_mapping");
request.patterns(Collections.singletonList("te*"));
request.patterns(singletonList("te*"));
request.putMapping("type1", "{}");
Set<Alias> aliases = new HashSet<>();
aliases.add(new Alias("invalid_alias").filter("abcde"));
@ -140,15 +143,63 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
public void testFindTemplates() throws Exception {
client().admin().indices().prepareDeleteTemplate("*").get(); // Delete all existing templates
putTemplateDetail(new PutRequest("test", "foo-1").patterns(Arrays.asList("foo-*")).order(1));
putTemplateDetail(new PutRequest("test", "foo-2").patterns(Arrays.asList("foo-*")).order(2));
putTemplateDetail(new PutRequest("test", "bar").patterns(Arrays.asList("bar-*")).order(between(0, 100)));
putTemplateDetail(new PutRequest("test", "foo-1").patterns(singletonList("foo-*")).order(1));
putTemplateDetail(new PutRequest("test", "foo-2").patterns(singletonList("foo-*")).order(2));
putTemplateDetail(new PutRequest("test", "bar").patterns(singletonList("bar-*")).order(between(0, 100)));
final ClusterState state = client().admin().cluster().prepareState().get().getState();
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234").stream()
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234", randomBoolean()).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("foo-2", "foo-1"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz").stream()
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz", randomBoolean()).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("bar"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz"), empty());
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz", randomBoolean()), empty());
}
public void testFindTemplatesWithHiddenIndices() throws Exception {
client().admin().indices().prepareDeleteTemplate("*").get(); // Delete all existing templates
putTemplateDetail(new PutRequest("testFindTemplatesWithHiddenIndices", "foo-1").patterns(singletonList("foo-*")).order(1));
putTemplateDetail(new PutRequest("testFindTemplatesWithHiddenIndices", "foo-2").patterns(singletonList("foo-*")).order(2));
putTemplateDetail(
new PutRequest("testFindTemplatesWithHiddenIndices", "bar").patterns(singletonList("bar-*")).order(between(0, 100)));
putTemplateDetail(new PutRequest("testFindTemplatesWithHiddenIndices", "global").patterns(singletonList("*")));
putTemplateDetail(new PutRequest("testFindTemplatesWithHiddenIndices", "sneaky-hidden")
.patterns(singletonList("sneaky*")).settings(Settings.builder().put("index.hidden", true).build()));
final ClusterState state = client().admin().cluster().prepareState().get().getState();
// hidden
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234", true).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("foo-2", "foo-1"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz", true).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("bar"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz", true), empty());
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "sneaky1", true).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("sneaky-hidden"));
// not hidden
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234", false).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("foo-2", "foo-1", "global"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz", false).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("bar", "global"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz", false).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("global"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "sneaky1", false).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), containsInAnyOrder("global", "sneaky-hidden"));
// unknown
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234", null).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("foo-2", "foo-1", "global"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz", null).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("bar", "global"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz", null).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("global"));
assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "sneaky1", null).stream()
.map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("sneaky-hidden"));
}
public void testPutGlobalTemplateWithIndexHiddenSetting() throws Exception {
List<Throwable> errors = putTemplateDetail(new PutRequest("testPutGlobalTemplateWithIndexHiddenSetting", "sneaky-hidden")
.patterns(singletonList("*")).settings(Settings.builder().put("index.hidden", true).build()));
assertThat(errors.size(), is(1));
assertThat(errors.get(0).getMessage(), containsString("global templates may not specify the setting index.hidden"));
}
private static List<Throwable> putTemplate(NamedXContentRegistry xContentRegistry, PutRequest request) {
@ -192,7 +243,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
null,
null,
new Environment(builder().put(Environment.PATH_HOME_SETTING.getKey(), createTempDir().toString()).build(), null),
null,
IndexScopedSettings.DEFAULT_SCOPED_SETTINGS,
null,
xContentRegistry(),
true);

36
server/src/test/java/org/elasticsearch/action/search/MultiSearchRequestTests.java
View File

@ -26,10 +26,14 @@ import org.elasticsearch.common.CheckedRunnable;
import org.elasticsearch.common.ParseField;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.bytes.BytesArray;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.logging.DeprecationLogger;
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentHelper;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.common.xcontent.json.JsonXContent;
import org.elasticsearch.index.query.MatchAllQueryBuilder;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.rest.RestRequest;
@ -45,6 +49,7 @@ import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import static java.util.Collections.singletonList;
import static org.elasticsearch.search.RandomSearchRequestGenerator.randomSearchRequest;
@ -322,6 +327,37 @@ public class MultiSearchRequestTests extends ESTestCase {
}
}
public void testWritingExpandWildcards() throws IOException {
assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), true, true, true, randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean()), "all");
assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), true, true, false, randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean()), "open,closed");
assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), true, false, true, randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean()), "open,hidden");
assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), true, false, false, randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean()), "open");
assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), false, true, true, randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean()), "closed,hidden");
assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), false, true, false, randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean()), "closed");
assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), false, false, true, randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean()), "hidden");
assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), false, false, false, randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean()), "none");
}
private void assertExpandWildcardsValue(IndicesOptions options, String expectedValue) throws IOException {
SearchRequest request = new SearchRequest();
request.indicesOptions(options);
try (XContentBuilder builder = JsonXContent.contentBuilder()) {
MultiSearchRequest.writeSearchRequestParams(request, builder);
Map<String, Object> map =
XContentHelper.convertToMap(XContentType.JSON.xContent(), BytesReference.bytes(builder).streamInput(), false);
final String value = (String) map.get("expand_wildcards");
assertEquals(expectedValue, value);
}
}
public void testEqualsAndHashcode() {
checkEqualsAndHashCode(createMultiSearchRequest(), MultiSearchRequestTests::copyRequest, MultiSearchRequestTests::mutate);
}

104
server/src/test/java/org/elasticsearch/action/support/IndicesOptionsTests.java
View File

@ -35,6 +35,7 @@ import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.test.EqualsHashCodeTestUtils;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
@ -46,6 +47,7 @@ import java.util.OptionalInt;
import static org.elasticsearch.test.VersionUtils.randomVersionBetween;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.Matchers.is;
public class IndicesOptionsTests extends ESTestCase {
@ -55,7 +57,7 @@ public class IndicesOptionsTests extends ESTestCase {
Version version = randomVersionBetween(random(), Version.V_7_0_0, null);
IndicesOptions indicesOptions = IndicesOptions.fromOptions(
randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
randomBoolean());
randomBoolean(), randomBoolean());
BytesStreamOutput output = new BytesStreamOutput();
output.setVersion(version);
@ -69,6 +71,11 @@ public class IndicesOptionsTests extends ESTestCase {
assertThat(indicesOptions2.allowNoIndices(), equalTo(indicesOptions.allowNoIndices()));
assertThat(indicesOptions2.expandWildcardsOpen(), equalTo(indicesOptions.expandWildcardsOpen()));
assertThat(indicesOptions2.expandWildcardsClosed(), equalTo(indicesOptions.expandWildcardsClosed()));
if (version.before(Version.V_7_7_0)) {
assertThat(indicesOptions2.expandWildcardsHidden(), is(true));
} else {
assertThat(indicesOptions2.expandWildcardsHidden(), equalTo(indicesOptions.expandWildcardsHidden()));
}
assertThat(indicesOptions2.forbidClosedIndices(), equalTo(indicesOptions.forbidClosedIndices()));
assertThat(indicesOptions2.allowAliasesToMultipleIndices(), equalTo(indicesOptions.allowAliasesToMultipleIndices()));
@ -110,27 +117,30 @@ public class IndicesOptionsTests extends ESTestCase {
}
public void testFromOptions() {
boolean ignoreUnavailable = randomBoolean();
boolean allowNoIndices = randomBoolean();
boolean expandToOpenIndices = randomBoolean();
boolean expandToClosedIndices = randomBoolean();
boolean allowAliasesToMultipleIndices = randomBoolean();
boolean forbidClosedIndices = randomBoolean();
boolean ignoreAliases = randomBoolean();
boolean ingoreThrottled = randomBoolean();
final boolean ignoreUnavailable = randomBoolean();
final boolean allowNoIndices = randomBoolean();
final boolean expandToOpenIndices = randomBoolean();
final boolean expandToClosedIndices = randomBoolean();
final boolean expandToHiddenIndices = randomBoolean();
final boolean allowAliasesToMultipleIndices = randomBoolean();
final boolean forbidClosedIndices = randomBoolean();
final boolean ignoreAliases = randomBoolean();
final boolean ignoreThrottled = randomBoolean();
IndicesOptions indicesOptions = IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices,expandToOpenIndices,
expandToClosedIndices, allowAliasesToMultipleIndices, forbidClosedIndices, ignoreAliases, ingoreThrottled);
IndicesOptions indicesOptions = IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices,
expandToClosedIndices, expandToHiddenIndices, allowAliasesToMultipleIndices, forbidClosedIndices, ignoreAliases,
ignoreThrottled);
assertThat(indicesOptions.ignoreUnavailable(), equalTo(ignoreUnavailable));
assertThat(indicesOptions.allowNoIndices(), equalTo(allowNoIndices));
assertThat(indicesOptions.expandWildcardsOpen(), equalTo(expandToOpenIndices));
assertThat(indicesOptions.expandWildcardsClosed(), equalTo(expandToClosedIndices));
assertThat(indicesOptions.expandWildcardsHidden(), equalTo(expandToHiddenIndices));
assertThat(indicesOptions.allowAliasesToMultipleIndices(), equalTo(allowAliasesToMultipleIndices));
assertThat(indicesOptions.allowAliasesToMultipleIndices(), equalTo(allowAliasesToMultipleIndices));
assertThat(indicesOptions.forbidClosedIndices(), equalTo(forbidClosedIndices));
assertEquals(ignoreAliases, indicesOptions.ignoreAliases());
assertEquals(ingoreThrottled, indicesOptions.ignoreThrottled());
assertEquals(ignoreThrottled, indicesOptions.ignoreThrottled());
}
public void testFromOptionsWithDefaultOptions() {
@ -139,53 +149,64 @@ public class IndicesOptionsTests extends ESTestCase {
boolean expandToOpenIndices = randomBoolean();
boolean expandToClosedIndices = randomBoolean();
IndicesOptions defaultOptions = IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(),
IndicesOptions defaultOptions = IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean());
IndicesOptions indicesOptions = IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices,expandToOpenIndices,
IndicesOptions indicesOptions = IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices,
expandToClosedIndices, defaultOptions);
assertEquals(ignoreUnavailable, indicesOptions.ignoreUnavailable());
assertEquals(allowNoIndices, indicesOptions.allowNoIndices());
assertEquals(expandToOpenIndices, indicesOptions.expandWildcardsOpen());
assertEquals(expandToClosedIndices, indicesOptions.expandWildcardsClosed());
assertEquals(defaultOptions.expandWildcardsHidden(), indicesOptions.expandWildcardsHidden());
assertEquals(defaultOptions.allowAliasesToMultipleIndices(), indicesOptions.allowAliasesToMultipleIndices());
assertEquals(defaultOptions.forbidClosedIndices(), indicesOptions.forbidClosedIndices());
assertEquals(defaultOptions.ignoreAliases(), indicesOptions.ignoreAliases());
}
public void testFromParameters() {
boolean expandWildcardsOpen = randomBoolean();
boolean expandWildcardsClosed = randomBoolean();
String expandWildcardsString;
if (expandWildcardsOpen && expandWildcardsClosed) {
final boolean expandWildcardsOpen = randomBoolean();
final boolean expandWildcardsClosed = randomBoolean();
final boolean expandWildcardsHidden = randomBoolean();
final String expandWildcardsString;
if (expandWildcardsOpen && expandWildcardsClosed && expandWildcardsHidden) {
if (randomBoolean()) {
expandWildcardsString = "open,closed";
expandWildcardsString = "open,closed,hidden";
} else {
expandWildcardsString = "all";
}
} else if (expandWildcardsOpen) {
expandWildcardsString = "open";
} else if (expandWildcardsClosed) {
expandWildcardsString = "closed";
} else {
expandWildcardsString = "none";
List<String> values = new ArrayList<>();
if (expandWildcardsOpen) {
values.add("open");
}
if (expandWildcardsClosed) {
values.add("closed");
}
if (expandWildcardsHidden) {
values.add("hidden");
}
if (values.isEmpty() && randomBoolean()) {
values.add("none");
}
expandWildcardsString = String.join(",", values);
}
boolean ignoreUnavailable = randomBoolean();
String ignoreUnavailableString = Boolean.toString(ignoreUnavailable);
boolean ignoreThrottled = randomBoolean();
String ignoreThrottledString = Boolean.toString(ignoreThrottled);
boolean allowNoIndices = randomBoolean();
String allowNoIndicesString = Boolean.toString(allowNoIndices);
IndicesOptions defaultOptions = IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean());
randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean());
IndicesOptions updatedOptions = IndicesOptions.fromParameters(expandWildcardsString, ignoreUnavailableString,
allowNoIndicesString, ignoreThrottled, defaultOptions);
assertEquals(expandWildcardsOpen, updatedOptions.expandWildcardsOpen());
assertEquals(expandWildcardsClosed, updatedOptions.expandWildcardsClosed());
assertEquals(expandWildcardsHidden, updatedOptions.expandWildcardsHidden());
assertEquals(ignoreUnavailable, updatedOptions.ignoreUnavailable());
assertEquals(allowNoIndices, updatedOptions.allowNoIndices());
assertEquals(defaultOptions.allowAliasesToMultipleIndices(), updatedOptions.allowAliasesToMultipleIndices());
@ -194,20 +215,20 @@ public class IndicesOptionsTests extends ESTestCase {
}
public void testEqualityAndHashCode() {
IndicesOptions indicesOptions = IndicesOptions.fromOptions(
randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
randomBoolean());
IndicesOptions indicesOptions = IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean());
EqualsHashCodeTestUtils.checkEqualsAndHashCode(indicesOptions, opts -> {
return IndicesOptions.fromOptions(opts.ignoreUnavailable(), opts.allowNoIndices(), opts.expandWildcardsOpen(),
opts.expandWildcardsClosed(), opts.allowAliasesToMultipleIndices(), opts.forbidClosedIndices(), opts.ignoreAliases(),
opts.ignoreThrottled());
opts.expandWildcardsClosed(), opts.expandWildcardsHidden(), opts.allowAliasesToMultipleIndices(),
opts.forbidClosedIndices(), opts.ignoreAliases(), opts.ignoreThrottled());
}, opts -> {
boolean mutated = false;
boolean ignoreUnavailable = opts.ignoreUnavailable();
boolean allowNoIndices = opts.allowNoIndices();
boolean expandOpen = opts.expandWildcardsOpen();
boolean expandClosed = opts.expandWildcardsClosed();
boolean expandHidden = opts.expandWildcardsHidden();
boolean allowAliasesToMulti = opts.allowAliasesToMultipleIndices();
boolean forbidClosed = opts.forbidClosedIndices();
boolean ignoreAliases = opts.ignoreAliases();
@ -229,6 +250,10 @@ public class IndicesOptionsTests extends ESTestCase {
expandClosed = !expandClosed;
mutated = true;
}
if (randomBoolean()) {
expandHidden = !expandHidden;
mutated = true;
}
if (randomBoolean()) {
allowAliasesToMulti = !allowAliasesToMulti;
mutated = true;
@ -246,7 +271,7 @@ public class IndicesOptionsTests extends ESTestCase {
mutated = true;
}
}
return IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices, expandOpen, expandClosed,
return IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices, expandOpen, expandClosed, expandHidden,
allowAliasesToMulti, forbidClosed, ignoreAliases, ignoreThrottled);
});
}
@ -254,7 +279,7 @@ public class IndicesOptionsTests extends ESTestCase {
public void testFromMap() {
IndicesOptions defaults = IndicesOptions.strictExpandOpen();
Collection<String> wildcardStates = randomBoolean() ?
null : randomSubsetOf(Arrays.asList("open", "closed"));
null : randomSubsetOf(Arrays.asList("open", "closed", "hidden"));
Boolean ignoreUnavailable = randomBoolean() ? null : randomBoolean();
Boolean allowNoIndices = randomBoolean() ? null : randomBoolean();
Boolean ignoreThrottled = randomBoolean() ? null : randomBoolean();
@ -280,13 +305,15 @@ public class IndicesOptionsTests extends ESTestCase {
IndicesOptions fromMap = IndicesOptions.fromMap(settings, defaults);
boolean open = wildcardStates != null ? wildcardStates.contains("open") : defaults.expandWildcardsOpen();
assertEquals(fromMap.expandWildcardsOpen(), open);
assertEquals(open, fromMap.expandWildcardsOpen());
boolean closed = wildcardStates != null ? wildcardStates.contains("closed") : defaults.expandWildcardsClosed();
assertEquals(fromMap.expandWildcardsClosed(), closed);
assertEquals(closed, fromMap.expandWildcardsClosed());
boolean hidden = wildcardStates != null ? wildcardStates.contains("hidden") : defaults.expandWildcardsHidden();
assertEquals(hidden, fromMap.expandWildcardsHidden());
assertEquals(fromMap.ignoreUnavailable(), ignoreUnavailable == null ? defaults.ignoreUnavailable() : ignoreUnavailable);
assertEquals(fromMap.allowNoIndices(), allowNoIndices == null ? defaults.allowNoIndices() : allowNoIndices);
assertEquals(fromMap.ignoreThrottled(), ignoreThrottled == null ? defaults.ignoreThrottled() : ignoreThrottled);
assertEquals(ignoreUnavailable == null ? defaults.ignoreUnavailable() : ignoreUnavailable, fromMap.ignoreUnavailable());
assertEquals(allowNoIndices == null ? defaults.allowNoIndices() : allowNoIndices, fromMap.allowNoIndices());
assertEquals(ignoreThrottled == null ? defaults.ignoreThrottled() : ignoreThrottled, fromMap.ignoreThrottled());
}
public void testToXContent() throws IOException {
@ -317,6 +344,7 @@ public class IndicesOptionsTests extends ESTestCase {
} else {
assertFalse(((List<?>) map.get("expand_wildcards")).contains("closed"));
}
assertEquals(wildcardStates.contains(WildcardStates.HIDDEN), ((List<?>) map.get("expand_wildcards")).contains("hidden"));
assertEquals(map.get("ignore_unavailable"), options.contains(Option.IGNORE_UNAVAILABLE));
assertEquals(map.get("allow_no_indices"), options.contains(Option.ALLOW_NO_INDICES));
assertEquals(map.get("ignore_throttled"), options.contains(Option.IGNORE_THROTTLED));

163
server/src/test/java/org/elasticsearch/cluster/metadata/IndexNameExpressionResolverTests.java
View File

@ -272,11 +272,15 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
MetaData.Builder mdBuilder = MetaData.builder()
.put(indexBuilder("foo").state(IndexMetaData.State.CLOSE))
.put(indexBuilder("bar"))
.put(indexBuilder("foobar").putAlias(AliasMetaData.builder("barbaz")));
.put(indexBuilder("foobar").putAlias(AliasMetaData.builder("barbaz")))
.put(indexBuilder("hidden", Settings.builder().put("index.hidden", true).build()))
.put(indexBuilder(".hidden", Settings.builder().put("index.hidden", true).build()))
.put(indexBuilder(".hidden-closed", Settings.builder().put("index.hidden", true).build()).state(IndexMetaData.State.CLOSE))
.put(indexBuilder("hidden-closed", Settings.builder().put("index.hidden", true).build()).state(IndexMetaData.State.CLOSE));
ClusterState state = ClusterState.builder(new ClusterName("_name")).metaData(mdBuilder).build();
// Only closed
IndicesOptions options = IndicesOptions.fromOptions(false, true, false, true);
IndicesOptions options = IndicesOptions.fromOptions(false, true, false, true, false);
IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, options);
String[] results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
assertEquals(1, results.length);
@ -291,8 +295,17 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
assertEquals(1, results.length);
assertEquals("bar", results[0]);
// implicit hidden for dot indices based on wildcard starting with .
results = indexNameExpressionResolver.concreteIndexNames(context, ".*");
assertEquals(1, results.length);
assertThat(results, arrayContainingInAnyOrder(".hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, ".hidd*");
assertEquals(1, results.length);
assertThat(results, arrayContainingInAnyOrder(".hidden-closed"));
// Only open
options = IndicesOptions.fromOptions(false, true, true, false);
options = IndicesOptions.fromOptions(false, true, true, false, false);
context = new IndexNameExpressionResolver.Context(state, options);
results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
assertEquals(2, results.length);
@ -306,8 +319,17 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
assertEquals(1, results.length);
assertEquals("bar", results[0]);
// implicit hidden for dot indices based on wildcard starting with .
results = indexNameExpressionResolver.concreteIndexNames(context, ".*");
assertEquals(1, results.length);
assertThat(results, arrayContainingInAnyOrder(".hidden"));
results = indexNameExpressionResolver.concreteIndexNames(context, ".hidd*");
assertEquals(1, results.length);
assertThat(results, arrayContainingInAnyOrder(".hidden"));
// Open and closed
options = IndicesOptions.fromOptions(false, true, true, true);
options = IndicesOptions.fromOptions(false, true, true, true, false);
context = new IndexNameExpressionResolver.Context(state, options);
results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
assertEquals(3, results.length);
@ -332,7 +354,111 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
results = indexNameExpressionResolver.concreteIndexNames(context, "-*");
assertEquals(0, results.length);
options = IndicesOptions.fromOptions(false, false, true, true);
// implicit hidden for dot indices based on wildcard starting with .
results = indexNameExpressionResolver.concreteIndexNames(context, ".*");
assertEquals(2, results.length);
assertThat(results, arrayContainingInAnyOrder(".hidden", ".hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, ".hidd*");
assertEquals(2, results.length);
assertThat(results, arrayContainingInAnyOrder(".hidden", ".hidden-closed"));
// open closed and hidden
options = IndicesOptions.fromOptions(false, true, true, true, true);
context = new IndexNameExpressionResolver.Context(state, options);
results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
assertEquals(7, results.length);
assertThat(results, arrayContainingInAnyOrder("bar", "foobar", "foo", "hidden", "hidden-closed", ".hidden", ".hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, "foo*");
assertEquals(2, results.length);
assertThat(results, arrayContainingInAnyOrder("foobar", "foo"));
results = indexNameExpressionResolver.concreteIndexNames(context, "bar");
assertEquals(1, results.length);
assertEquals("bar", results[0]);
results = indexNameExpressionResolver.concreteIndexNames(context, "*", "-foo*");
assertEquals(5, results.length);
assertThat(results, arrayContainingInAnyOrder("bar", "hidden", "hidden-closed", ".hidden", ".hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, "*", "-foo", "-foobar");
assertEquals(5, results.length);
assertThat(results, arrayContainingInAnyOrder("bar", "hidden", "hidden-closed", ".hidden", ".hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, "*", "-foo", "-foobar", "-hidden*");
assertEquals(3, results.length);
assertThat(results, arrayContainingInAnyOrder("bar", ".hidden", ".hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, "hidden*");
assertEquals(2, results.length);
assertThat(results, arrayContainingInAnyOrder("hidden", "hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, "hidden");
assertEquals(1, results.length);
assertThat(results, arrayContainingInAnyOrder("hidden"));
results = indexNameExpressionResolver.concreteIndexNames(context, "hidden-closed");
assertEquals(1, results.length);
assertThat(results, arrayContainingInAnyOrder("hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, "-*");
assertEquals(0, results.length);
// open and hidden
options = IndicesOptions.fromOptions(false, true, true, false, true);
context = new IndexNameExpressionResolver.Context(state, options);
results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
assertEquals(4, results.length);
assertThat(results, arrayContainingInAnyOrder("bar", "foobar", "hidden", ".hidden"));
results = indexNameExpressionResolver.concreteIndexNames(context, "foo*");
assertEquals(1, results.length);
assertEquals("foobar", results[0]);
results = indexNameExpressionResolver.concreteIndexNames(context, "bar");
assertEquals(1, results.length);
assertEquals("bar", results[0]);
results = indexNameExpressionResolver.concreteIndexNames(context, "h*");
assertEquals(1, results.length);
assertEquals("hidden", results[0]);
// closed and hidden
options = IndicesOptions.fromOptions(false, true, false, true, true);
context = new IndexNameExpressionResolver.Context(state, options);
results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
assertEquals(3, results.length);
assertThat(results, arrayContainingInAnyOrder("foo", "hidden-closed", ".hidden-closed"));
results = indexNameExpressionResolver.concreteIndexNames(context, "foo*");
assertEquals(1, results.length);
assertEquals("foo", results[0]);
results = indexNameExpressionResolver.concreteIndexNames(context, "bar");
assertEquals(1, results.length);
assertEquals("bar", results[0]);
results = indexNameExpressionResolver.concreteIndexNames(context, "h*");
assertEquals(1, results.length);
assertEquals("hidden-closed", results[0]);
// only hidden
options = IndicesOptions.fromOptions(false, true, false, false, true);
context = new IndexNameExpressionResolver.Context(state, options);
results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
assertThat(results, emptyArray());
results = indexNameExpressionResolver.concreteIndexNames(context, "h*");
assertThat(results, emptyArray());
results = indexNameExpressionResolver.concreteIndexNames(context, "hidden");
assertThat(results, arrayContainingInAnyOrder("hidden"));
results = indexNameExpressionResolver.concreteIndexNames(context, "hidden-closed");
assertThat(results, arrayContainingInAnyOrder("hidden-closed"));
options = IndicesOptions.fromOptions(false, false, true, true, true);
IndexNameExpressionResolver.Context context2 = new IndexNameExpressionResolver.Context(state, options);
IndexNotFoundException infe = expectThrows(IndexNotFoundException.class,
() -> indexNameExpressionResolver.concreteIndexNames(context2, "-*"));
@ -349,7 +475,7 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
//ignore unavailable and allow no indices
{
IndicesOptions noExpandLenient = IndicesOptions.fromOptions(true, true, false, false);
IndicesOptions noExpandLenient = IndicesOptions.fromOptions(true, true, false, false, randomBoolean());
IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, noExpandLenient);
String[] results = indexNameExpressionResolver.concreteIndexNames(context, "baz*");
assertThat(results, emptyArray());
@ -371,7 +497,7 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
//ignore unavailable but don't allow no indices
{
IndicesOptions noExpandDisallowEmpty = IndicesOptions.fromOptions(true, false, false, false);
IndicesOptions noExpandDisallowEmpty = IndicesOptions.fromOptions(true, false, false, false, randomBoolean());
IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, noExpandDisallowEmpty);
{
@ -396,7 +522,7 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
//error on unavailable but allow no indices
{
IndicesOptions noExpandErrorUnavailable = IndicesOptions.fromOptions(false, true, false, false);
IndicesOptions noExpandErrorUnavailable = IndicesOptions.fromOptions(false, true, false, false, randomBoolean());
IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, noExpandErrorUnavailable);
{
String[] results = indexNameExpressionResolver.concreteIndexNames(context, "baz*");
@ -422,7 +548,7 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
//error on both unavailable and no indices
{
IndicesOptions noExpandStrict = IndicesOptions.fromOptions(false, false, false, false);
IndicesOptions noExpandStrict = IndicesOptions.fromOptions(false, false, false, false, randomBoolean());
IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, noExpandStrict);
IndexNotFoundException infe = expectThrows(IndexNotFoundException.class,
() -> indexNameExpressionResolver.concreteIndexNames(context, "baz*"));
@ -539,12 +665,17 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
}
private static IndexMetaData.Builder indexBuilder(String index) {
return IndexMetaData.builder(index).settings(settings());
return indexBuilder(index, Settings.EMPTY);
}
private static Settings.Builder settings() {
private static IndexMetaData.Builder indexBuilder(String index, Settings additionalSettings) {
return IndexMetaData.builder(index).settings(settings(additionalSettings));
}
private static Settings.Builder settings(Settings additionalSettings) {
return settings(Version.CURRENT).put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, 1)
.put(IndexMetaData.SETTING_NUMBER_OF_REPLICAS, 0);
.put(IndexMetaData.SETTING_NUMBER_OF_REPLICAS, 0)
.put(additionalSettings);
}
public void testConcreteIndicesIgnoreIndicesOneMissingIndex() {
@ -1397,13 +1528,13 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
public void testIgnoreThrottled() {
MetaData.Builder mdBuilder = MetaData.builder()
.put(indexBuilder("test-index").state(State.OPEN)
.settings(settings().put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), true))
.put(indexBuilder("test-index", Settings.builder().put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), true).build())
.state(State.OPEN)
.putAlias(AliasMetaData.builder("test-alias")))
.put(indexBuilder("index").state(State.OPEN)
.putAlias(AliasMetaData.builder("test-alias2")))
.put(indexBuilder("index-closed").state(State.CLOSE)
.settings(settings().put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), true))
.put(indexBuilder("index-closed", Settings.builder().put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), true).build())
.state(State.CLOSE)
.putAlias(AliasMetaData.builder("test-alias-closed")));
ClusterState state = ClusterState.builder(new ClusterName("_name")).metaData(mdBuilder).build();
{

129
server/src/test/java/org/elasticsearch/index/HiddenIndexIT.java Normal file
View File

@ -0,0 +1,129 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.index;
import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse;
import org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.support.IndicesOptions;
import org.elasticsearch.action.support.WriteRequest.RefreshPolicy;
import org.elasticsearch.cluster.metadata.MappingMetaData;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.indices.InvalidIndexTemplateException;
import org.elasticsearch.test.ESIntegTestCase;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
public class HiddenIndexIT extends ESIntegTestCase {
public void testHiddenIndexSearch() {
assertAcked(client().admin().indices().prepareCreate("hidden-index")
.setSettings(Settings.builder().put("index.hidden", true).build())
.get());
client().prepareIndex("hidden-index", "_doc").setSource("foo", "bar").setRefreshPolicy(RefreshPolicy.IMMEDIATE).get();
// default not visible to wildcard expansion
SearchResponse searchResponse =
client().prepareSearch(randomFrom("*", "_all", "h*", "*index")).setSize(1000).setQuery(QueryBuilders.matchAllQuery()).get();
boolean matchedHidden = Arrays.stream(searchResponse.getHits().getHits()).anyMatch(hit -> "hidden-index".equals(hit.getIndex()));
assertFalse(matchedHidden);
// direct access allowed
searchResponse = client().prepareSearch("hidden-index").setSize(1000).setQuery(QueryBuilders.matchAllQuery()).get();
matchedHidden = Arrays.stream(searchResponse.getHits().getHits()).anyMatch(hit -> "hidden-index".equals(hit.getIndex()));
assertTrue(matchedHidden);
// with indices option to include hidden
searchResponse = client().prepareSearch(randomFrom("*", "_all", "h*", "*index"))
.setSize(1000)
.setQuery(QueryBuilders.matchAllQuery())
.setIndicesOptions(IndicesOptions.LENIENT_EXPAND_OPEN_HIDDEN)
.get();
matchedHidden = Arrays.stream(searchResponse.getHits().getHits()).anyMatch(hit -> "hidden-index".equals(hit.getIndex()));
assertTrue(matchedHidden);
// implicit based on use of pattern starting with . and a wildcard
assertAcked(client().admin().indices().prepareCreate(".hidden-index")
.setSettings(Settings.builder().put("index.hidden", true).build())
.get());
client().prepareIndex(".hidden-index", "_doc").setSource("foo", "bar").setRefreshPolicy(RefreshPolicy.IMMEDIATE).get();
searchResponse = client().prepareSearch(randomFrom(".*", ".hidden-*"))
.setSize(1000)
.setQuery(QueryBuilders.matchAllQuery())
.get();
matchedHidden = Arrays.stream(searchResponse.getHits().getHits()).anyMatch(hit -> ".hidden-index".equals(hit.getIndex()));
assertTrue(matchedHidden);
}
public void testGlobalTemplatesDoNotApply() {
assertAcked(client().admin().indices().preparePutTemplate("a_global_template").setPatterns(Collections.singletonList("*"))
.addMapping("_doc", "foo", "type=text").get());
assertAcked(client().admin().indices().preparePutTemplate("not_global_template").setPatterns(Collections.singletonList("a*"))
.addMapping("_doc", "bar", "type=text").get());
assertAcked(client().admin().indices().preparePutTemplate("specific_template")
.setPatterns(Collections.singletonList("a_hidden_index")).addMapping("_doc", "baz", "type=text").get());
assertAcked(client().admin().indices().preparePutTemplate("unused_template").setPatterns(Collections.singletonList("not_used"))
.addMapping("_doc", "foobar", "type=text").get());
assertAcked(client().admin().indices().prepareCreate("a_hidden_index")
.setSettings(Settings.builder().put("index.hidden", true).build()).get());
GetMappingsResponse mappingsResponse = client().admin().indices().prepareGetMappings("a_hidden_index").get();
assertThat(mappingsResponse.mappings().size(), is(1));
MappingMetaData mappingMetaData = mappingsResponse.mappings().get("a_hidden_index").get("_doc");
assertNotNull(mappingMetaData);
Map<String, Object> propertiesMap = (Map<String, Object>) mappingMetaData.getSourceAsMap().get("properties");
assertNotNull(propertiesMap);
assertThat(propertiesMap.size(), is(2));
Map<String, Object> barMap = (Map<String, Object>) propertiesMap.get("bar");
assertNotNull(barMap);
assertThat(barMap.get("type"), is("text"));
Map<String, Object> bazMap = (Map<String, Object>) propertiesMap.get("baz");
assertNotNull(bazMap);
assertThat(bazMap.get("type"), is("text"));
}
public void testGlobalTemplateCannotMakeIndexHidden() {
InvalidIndexTemplateException invalidIndexTemplateException = expectThrows(InvalidIndexTemplateException.class,
() -> client().admin().indices().preparePutTemplate("a_global_template")
.setPatterns(Collections.singletonList("*"))
.setSettings(Settings.builder().put("index.hidden", randomBoolean()).build())
.get());
assertThat(invalidIndexTemplateException.getMessage(), containsString("global templates may not specify the setting index.hidden"));
}
public void testNonGlobalTemplateCanMakeIndexHidden() {
assertAcked(client().admin().indices().preparePutTemplate("a_global_template")
.setPatterns(Collections.singletonList("my_hidden_pattern*"))
.addMapping("_doc", "foo", "type=text")
.setSettings(Settings.builder().put("index.hidden", true).build())
.get());
assertAcked(client().admin().indices().prepareCreate("my_hidden_pattern1").get());
GetSettingsResponse getSettingsResponse = client().admin().indices().prepareGetSettings("my_hidden_pattern1").get();
assertThat(getSettingsResponse.getSetting("my_hidden_pattern1", "index.hidden"), is("true"));
}
}

5
test/framework/src/main/java/org/elasticsearch/test/TestCluster.java
View File

@ -25,6 +25,7 @@ import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
import org.elasticsearch.action.support.IndicesOptions;
import org.elasticsearch.client.Client;
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
@ -139,7 +140,9 @@ public abstract class TestCluster implements Closeable {
assert indices != null && indices.length > 0;
if (size() > 0) {
try {
assertAcked(client().admin().indices().prepareDelete(indices));
// include wiping hidden indices!
assertAcked(client().admin().indices().prepareDelete(indices)
.setIndicesOptions(IndicesOptions.fromOptions(false, true, true, true, true, false, false, true, false)));
} catch (IndexNotFoundException e) {
// ignore
} catch (IllegalArgumentException e) {

1
x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/job/persistence/JobResultsProviderTests.java
View File

@ -163,6 +163,7 @@ public class JobResultsProviderTests extends ESTestCase {
ImmutableOpenMap<String, AliasMetaData> aliases = ImmutableOpenMap.of();
when(indexMetaData.getAliases()).thenReturn(aliases);
when(indexMetaData.getSettings()).thenReturn(Settings.EMPTY);
ImmutableOpenMap<String, IndexMetaData> indexMap = ImmutableOpenMap.<String, IndexMetaData>builder()
.fPut(AnomalyDetectorsIndex.jobResultsAliasedName("foo"), indexMetaData).build();

41
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java
View File

@ -17,6 +17,7 @@ import org.elasticsearch.action.support.IndicesOptions;
import org.elasticsearch.cluster.metadata.AliasMetaData;
import org.elasticsearch.cluster.metadata.AliasOrIndex;
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.cluster.metadata.IndexMetaData.State;
import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver;
import org.elasticsearch.cluster.metadata.MetaData;
import org.elasticsearch.cluster.service.ClusterService;
@ -121,22 +122,21 @@ class IndicesAndAliasesResolver {
: "indices are: " + Arrays.toString(indicesRequest.indices()); // Arrays.toString() can handle null values - all good
resolvedIndicesBuilder.addLocal(getPutMappingIndexOrAlias((PutMappingRequest) indicesRequest, authorizedIndices, metaData));
} else if (indicesRequest instanceof IndicesRequest.Replaceable) {
IndicesRequest.Replaceable replaceable = (IndicesRequest.Replaceable) indicesRequest;
final boolean replaceWildcards = indicesRequest.indicesOptions().expandWildcardsOpen()
|| indicesRequest.indicesOptions().expandWildcardsClosed();
final IndicesRequest.Replaceable replaceable = (IndicesRequest.Replaceable) indicesRequest;
IndicesOptions indicesOptions = indicesRequest.indicesOptions();
final boolean replaceWildcards = indicesOptions.expandWildcardsOpen() || indicesOptions.expandWildcardsClosed();
if (indicesRequest instanceof IndicesExistsRequest) {
//indices exists api should never throw exception, make sure that ignore_unavailable and allow_no_indices are true
//we have to mimic what TransportIndicesExistsAction#checkBlock does in es core
indicesOptions = IndicesOptions.fromOptions(true, true,
indicesOptions.expandWildcardsOpen(), indicesOptions.expandWildcardsClosed());
indicesOptions.expandWildcardsOpen(), indicesOptions.expandWildcardsClosed());
}
// check for all and return list of authorized indices
if (IndexNameExpressionResolver.isAllIndices(indicesList(indicesRequest.indices()))) {
if (replaceWildcards) {
for (String authorizedIndex : authorizedIndices) {
if (isIndexVisible(authorizedIndex, indicesOptions, metaData)) {
if (isIndexVisible("*", authorizedIndex, indicesOptions, metaData)) {
resolvedIndicesBuilder.addLocal(authorizedIndex);
}
}
@ -363,7 +363,8 @@ class IndicesAndAliasesResolver {
if (replaceWildcards && Regex.isSimpleMatchPattern(dateMathName)) {
// continue
aliasOrIndex = dateMathName;
} else if (authorizedIndices.contains(dateMathName) && isIndexVisible(dateMathName, indicesOptions, metaData, true)) {
} else if (authorizedIndices.contains(dateMathName) &&
isIndexVisible(aliasOrIndex, dateMathName, indicesOptions, metaData, true)) {
if (minus) {
finalIndices.remove(dateMathName);
} else {
@ -380,7 +381,8 @@ class IndicesAndAliasesResolver {
wildcardSeen = true;
Set<String> resolvedIndices = new HashSet<>();
for (String authorizedIndex : authorizedIndices) {
if (Regex.simpleMatch(aliasOrIndex, authorizedIndex) && isIndexVisible(authorizedIndex, indicesOptions, metaData)) {
if (Regex.simpleMatch(aliasOrIndex, authorizedIndex) &&
isIndexVisible(aliasOrIndex, authorizedIndex, indicesOptions, metaData)) {
resolvedIndices.add(authorizedIndex);
}
}
@ -415,11 +417,12 @@ class IndicesAndAliasesResolver {
return finalIndices;
}
private static boolean isIndexVisible(String index, IndicesOptions indicesOptions, MetaData metaData) {
return isIndexVisible(index, indicesOptions, metaData, false);
private static boolean isIndexVisible(String expression, String index, IndicesOptions indicesOptions, MetaData metaData) {
return isIndexVisible(expression, index, indicesOptions, metaData, false);
}
private static boolean isIndexVisible(String index, IndicesOptions indicesOptions, MetaData metaData, boolean dateMathExpression) {
private static boolean isIndexVisible(String expression, String index, IndicesOptions indicesOptions, MetaData metaData,
boolean dateMathExpression) {
AliasOrIndex aliasOrIndex = metaData.getAliasAndIndexLookup().get(index);
if (aliasOrIndex.isAlias()) {
//it's an alias, ignore expandWildcardsOpen and expandWildcardsClosed.
@ -429,15 +432,29 @@ class IndicesAndAliasesResolver {
}
assert aliasOrIndex.getIndices().size() == 1 : "concrete index must point to a single index";
IndexMetaData indexMetaData = aliasOrIndex.getIndices().get(0);
if (indexMetaData.getState() == IndexMetaData.State.CLOSE && (indicesOptions.expandWildcardsClosed() || dateMathExpression)) {
final boolean isHidden = IndexMetaData.INDEX_HIDDEN_SETTING.get(indexMetaData.getSettings());
if (isHidden && indicesOptions.expandWildcardsHidden() == false && isVisibleDueToImplicitHidden(expression, index) == false) {
return false;
}
// the index is not hidden and since it is a date math expression, we consider it visible regardless of open/closed
if (dateMathExpression) {
assert State.values().length == 2 : "a new IndexMetaData.State value may need to be handled!";
return true;
}
if (indexMetaData.getState() == IndexMetaData.State.OPEN && (indicesOptions.expandWildcardsOpen() || dateMathExpression)) {
if (indexMetaData.getState() == IndexMetaData.State.CLOSE && indicesOptions.expandWildcardsClosed()) {
return true;
}
if (indexMetaData.getState() == IndexMetaData.State.OPEN && indicesOptions.expandWildcardsOpen()) {
return true;
}
return false;
}
private static boolean isVisibleDueToImplicitHidden(String expression, String index) {
return index.startsWith(".") && expression.startsWith(".") && Regex.isSimpleMatchPattern(expression);
}
private static List<String> indicesList(String[] list) {
return (list == null) ? null : Arrays.asList(list);
}

68
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java
View File

@ -35,6 +35,7 @@ import org.elasticsearch.action.support.PlainActionFuture;
import org.elasticsearch.action.termvectors.MultiTermVectorsRequest;
import org.elasticsearch.cluster.metadata.AliasMetaData;
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.cluster.metadata.IndexMetaData.State;
import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver;
import org.elasticsearch.cluster.metadata.MetaData;
import org.elasticsearch.cluster.service.ClusterService;
@ -125,13 +126,13 @@ public class IndicesAndAliasesResolverTests extends ESTestCase {
.putAlias(AliasMetaData.builder("foounauthorized")).settings(settings))
.put(indexBuilder("foobar").putAlias(AliasMetaData.builder("foofoobar"))
.putAlias(AliasMetaData.builder("foobarfoo")).settings(settings))
.put(indexBuilder("closed").state(IndexMetaData.State.CLOSE)
.put(indexBuilder("closed").state(State.CLOSE)
.putAlias(AliasMetaData.builder("foofoobar")).settings(settings))
.put(indexBuilder("foofoo-closed").state(IndexMetaData.State.CLOSE).settings(settings))
.put(indexBuilder("foobar-closed").state(IndexMetaData.State.CLOSE).settings(settings))
.put(indexBuilder("foofoo-closed").state(State.CLOSE).settings(settings))
.put(indexBuilder("foobar-closed").state(State.CLOSE).settings(settings))
.put(indexBuilder("foofoo").putAlias(AliasMetaData.builder("barbaz")).settings(settings))
.put(indexBuilder("bar").settings(settings))
.put(indexBuilder("bar-closed").state(IndexMetaData.State.CLOSE).settings(settings))
.put(indexBuilder("bar-closed").state(State.CLOSE).settings(settings))
.put(indexBuilder("bar2").settings(settings))
.put(indexBuilder(indexNameExpressionResolver.resolveDateMathExpression("<datetime-{now/M}>")).settings(settings))
.put(indexBuilder("-index10").settings(settings))
@ -141,6 +142,12 @@ public class IndicesAndAliasesResolverTests extends ESTestCase {
.put(indexBuilder("logs-00001").putAlias(AliasMetaData.builder("logs-alias").writeIndex(false)).settings(settings))
.put(indexBuilder("logs-00002").putAlias(AliasMetaData.builder("logs-alias").writeIndex(false)).settings(settings))
.put(indexBuilder("logs-00003").putAlias(AliasMetaData.builder("logs-alias").writeIndex(true)).settings(settings))
.put(indexBuilder("hidden-open").settings(Settings.builder().put(settings).put("index.hidden", true).build()))
.put(indexBuilder(".hidden-open").settings(Settings.builder().put(settings).put("index.hidden", true).build()))
.put(indexBuilder(".hidden-closed").state(State.CLOSE)
.settings(Settings.builder().put(settings).put("index.hidden", true).build()))
.put(indexBuilder("hidden-closed").state(State.CLOSE)
.settings(Settings.builder().put(settings).put("index.hidden", true).build()))
.put(indexBuilder(securityIndexName).settings(settings)).build();
if (withAlias) {
@ -152,7 +159,8 @@ public class IndicesAndAliasesResolverTests extends ESTestCase {
userDashIndices = new User("dash", "dash");
userNoIndices = new User("test", "test");
rolesStore = mock(CompositeRolesStore.class);
String[] authorizedIndices = new String[] { "bar", "bar-closed", "foofoobar", "foobarfoo", "foofoo", "missing", "foofoo-closed"};
String[] authorizedIndices = new String[] { "bar", "bar-closed", "foofoobar", "foobarfoo", "foofoo", "missing", "foofoo-closed",
"hidden-open", "hidden-closed", ".hidden-open", ".hidden-closed"};
String[] dashIndices = new String[]{"-index10", "-index11", "-index20", "-index21"};
roleMap = new HashMap<>();
roleMap.put("role", new RoleDescriptor("role", null,
@ -1392,8 +1400,56 @@ public class IndicesAndAliasesResolverTests extends ESTestCase {
assertEquals(message, index, putMappingIndexOrAlias);
}
// TODO with the removal of DeleteByQuery is there another way to test resolving a write action?
public void testHiddenIndicesResolution() {
SearchRequest searchRequest = new SearchRequest();
searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, true, true, true));
List<String> authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
ResolvedIndices resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
assertThat(resolvedIndices.getLocal(), containsInAnyOrder("bar", "bar-closed", "foofoobar", "foobarfoo", "foofoo", "foofoo-closed",
"hidden-open", "hidden-closed", ".hidden-open", ".hidden-closed"));
assertThat(resolvedIndices.getRemote(), emptyIterable());
// open + hidden
searchRequest = new SearchRequest();
searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, true, false, true));
authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
assertThat(resolvedIndices.getLocal(),
containsInAnyOrder("bar", "foofoobar", "foobarfoo", "foofoo", "hidden-open", ".hidden-open"));
assertThat(resolvedIndices.getRemote(), emptyIterable());
// open + implicit hidden for . indices
searchRequest = new SearchRequest(randomFrom(".*", ".hid*"));
searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, true, false, false));
authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
assertThat(resolvedIndices.getLocal(), containsInAnyOrder(".hidden-open"));
assertThat(resolvedIndices.getRemote(), emptyIterable());
// closed + hidden, ignore aliases
searchRequest = new SearchRequest();
searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, false, true, true, true, false, true, false));
authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
assertThat(resolvedIndices.getLocal(), containsInAnyOrder("bar-closed", "foofoo-closed", "hidden-closed", ".hidden-closed"));
assertThat(resolvedIndices.getRemote(), emptyIterable());
// closed + implicit hidden for . indices
searchRequest = new SearchRequest(randomFrom(".*", ".hid*"));
searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, false, true, false));
authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
assertThat(resolvedIndices.getLocal(), containsInAnyOrder(".hidden-closed"));
assertThat(resolvedIndices.getRemote(), emptyIterable());
// allow no indices, do not expand to open or closed, expand hidden, ignore aliases
searchRequest = new SearchRequest();
searchRequest.indicesOptions(IndicesOptions.fromOptions(false, true, false, false, false, true, false, true, false));
authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
assertThat(resolvedIndices.getLocal(), contains("-*"));
assertThat(resolvedIndices.getRemote(), emptyIterable());
}
private List<String> buildAuthorizedIndices(User user, String action) {
PlainActionFuture<Role> rolesListener = new PlainActionFuture<>();