Removed dynamic index names in favour for ES' date math index names.
Original commit: elastic/x-pack-elasticsearch@267084f163
This commit is contained in:
parent
7fb98b60b4
commit
e16894fb9d
|
@ -423,5 +423,3 @@ name in the `file` field. For example, the following snippet references the scri
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
||||||
include::how-watcher-works/dynamic-index-names.asciidoc[]
|
|
||||||
|
|
|
@ -1,106 +0,0 @@
|
||||||
[[dynamic-index-names]]
|
|
||||||
=== Dynamic Index Names
|
|
||||||
|
|
||||||
Several watch constructs deal with indices, including <<actions-index, `index` action>>,
|
|
||||||
the <<transform-search, `search` transform>> and the <<input-search, `search` input>>.
|
|
||||||
When configuring these constructs you can set the index names to static values. In addition
|
|
||||||
to specifying static index names, Watcher enables you to specify indexes using dynamic
|
|
||||||
time-aware templates. These templates resolve to specific index names during the watch
|
|
||||||
execution according to the execution time.
|
|
||||||
|
|
||||||
Dynamic index name resolution enables you to search a range of time-series indices, rather
|
|
||||||
than searching all of your time-series indices and filtering the the results. Limiting the
|
|
||||||
number of indices that are searched reduces the load on the cluster and improves watch
|
|
||||||
execution performance. For example, if you are using a watch to monitor errors in your
|
|
||||||
daily logs, you can use a dynamic index name template to restrict the search to the past
|
|
||||||
two days.
|
|
||||||
|
|
||||||
A dynamic index name takes the following form:
|
|
||||||
|
|
||||||
[source,txt]
|
|
||||||
----------------------------------------------------------------------
|
|
||||||
<static_name{date_math_expr{date_format}}>
|
|
||||||
----------------------------------------------------------------------
|
|
||||||
|
|
||||||
Where:
|
|
||||||
|
|
||||||
* `static_name` is the static text part of the name
|
|
||||||
* `date_math_expr` is a dynamic date math expression that computes the date dynamically
|
|
||||||
* `date_format` is the format in which the computed date should be rendered
|
|
||||||
|
|
||||||
NOTE: You must enclose dynamic index name templates within angle brackets. For example,
|
|
||||||
`<logstash-{now/d-2d}>`
|
|
||||||
|
|
||||||
The following example shows different forms of dynamic index names and the final index names
|
|
||||||
they resolve to given the execution date is 22rd March 2024.
|
|
||||||
|
|
||||||
[options="header"]
|
|
||||||
|======
|
|
||||||
| Expression |Resolves to
|
|
||||||
| `<logstash-{now/d}>` | `logstash-2024.03.22`
|
|
||||||
| `<logstash-{now/M}>` | `logstash-2024.03.01`
|
|
||||||
| `<logstash-{now/M{YYYY.MM}}>` | `logstash-2024.03`
|
|
||||||
| `<logstash-{now/M-1M{YYYY.MM}}>` | `logstash-2024.02`
|
|
||||||
|======
|
|
||||||
|
|
||||||
To use the characters `{` and `}` in the static part of an index name template, escape them
|
|
||||||
with a backslash, `\`:
|
|
||||||
|
|
||||||
* `<elastic\\{ON\\}-{now/M}>` resolves to `elastic{ON}-2024.03.01`
|
|
||||||
|
|
||||||
The following example shows a search input that searches the Logstash indices for the past
|
|
||||||
three days, assuming the indices use the default Logstash index name format,
|
|
||||||
`logstash-YYYY.MM.dd`.
|
|
||||||
|
|
||||||
[source,json]
|
|
||||||
----------------------------------------------------------------------
|
|
||||||
{
|
|
||||||
...
|
|
||||||
"input" : {
|
|
||||||
"search" : {
|
|
||||||
"request" : {
|
|
||||||
"indices" : [
|
|
||||||
"<logstash-{now/d-2d}>",
|
|
||||||
"<logstash-{now/d-1d}>",
|
|
||||||
"<logstash-{now/d}>"
|
|
||||||
],
|
|
||||||
...
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
...
|
|
||||||
}
|
|
||||||
----------------------------------------------------------------------
|
|
||||||
|
|
||||||
[[dynamic-index-name-timezone]]
|
|
||||||
|
|
||||||
By default, the index names are resolved base on `UTC` time zone. You can change this default at
|
|
||||||
multiple levels:
|
|
||||||
|
|
||||||
Configuring the following setting set the default dynamic index name time zone in watcher:
|
|
||||||
|
|
||||||
[source,yaml]
|
|
||||||
--------------------------------------------------
|
|
||||||
watcher.dynamic_indices.time_zone: '+01:00'
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
You can also configure the default time zone separately on each of the construct that make
|
|
||||||
use of it (`search` input/transform and `index` action):
|
|
||||||
|
|
||||||
[source,yaml]
|
|
||||||
--------------------------------------------------
|
|
||||||
watcher.input.search.dynamic_indices.time_zone: '+01:00'
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
[source,yaml]
|
|
||||||
--------------------------------------------------
|
|
||||||
watcher.transform.search.dynamic_indices.time_zone: '+01:00'
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
[source,yaml]
|
|
||||||
--------------------------------------------------
|
|
||||||
watcher.actions.index.dynamic_indices.time_zone: '+01:00'
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
Alternatively, each of these construct can define their own time zone within the watch
|
|
||||||
definition.
|
|
|
@ -33,7 +33,7 @@ The following snippet shows a simple `index` action definition:
|
||||||
|Name |Required | Default | Description
|
|Name |Required | Default | Description
|
||||||
|
|
||||||
| `index` | yes | - | The Elasticsearch index to
|
| `index` | yes | - | The Elasticsearch index to
|
||||||
index into. <<dynamic-index-names, Dynamic index names>>
|
index into.
|
||||||
are supported
|
are supported
|
||||||
|
|
||||||
| `doc_type` | yes | - | The type of the document
|
| `doc_type` | yes | - | The type of the document
|
||||||
|
@ -51,10 +51,6 @@ The following snippet shows a simple `index` action definition:
|
||||||
the default internal index/bulk operations
|
the default internal index/bulk operations
|
||||||
<<default-internal-ops-timeouts, timeouts>>.
|
<<default-internal-ops-timeouts, timeouts>>.
|
||||||
|
|
||||||
| `dynamic_name_timezone` | no | - | The time zone to use for resolving the index name based on
|
|
||||||
<<dynamic-index-names, Dynamic Index Names>>. The default
|
|
||||||
time zone also can be <<dynamic-index-name-timezone, configured>>
|
|
||||||
globally.
|
|
||||||
|======
|
|======
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -19,7 +19,7 @@ example, to get the message field from the first hit, use `ctx.payload.hits.hits
|
||||||
|======
|
|======
|
||||||
| Name |Required | Default | Description
|
| Name |Required | Default | Description
|
||||||
| `request.search_type` | no | count | The {ref}/search-request-search-type.html#search-request-search-type[type] of search request to perform. Valid values are: `count`, `dfs_query_and_fetch`, `dfs_query_then_fetch`, `query_and_fetch`, `query_then_fetch`, and `scan`. The Elasticsearch default is `query_then_fetch`.
|
| `request.search_type` | no | count | The {ref}/search-request-search-type.html#search-request-search-type[type] of search request to perform. Valid values are: `count`, `dfs_query_and_fetch`, `dfs_query_then_fetch`, `query_and_fetch`, `query_then_fetch`, and `scan`. The Elasticsearch default is `query_then_fetch`.
|
||||||
| `request.indices` | no | - | The indices to search. If omitted, all indices are searched, which is the default behaviour in Elasticsearch. <<dynamic-index-names, Dynamic index names>> are supported.
|
| `request.indices` | no | - | The indices to search. If omitted, all indices are searched, which is the default behaviour in Elasticsearch.
|
||||||
| `request.types` | no | - | The document types to search for. If omitted, all document types are are searched, which is the default behaviour in Elasticsearch.
|
| `request.types` | no | - | The document types to search for. If omitted, all document types are are searched, which is the default behaviour in Elasticsearch.
|
||||||
| `request.body` | no | - | The body of the request. The {ref}/search-request-body.html[request body] follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` <<templates, templates>>.
|
| `request.body` | no | - | The body of the request. The {ref}/search-request-body.html[request body] follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` <<templates, templates>>.
|
||||||
| `request.template` | no | - | The body of the search template. See <<templates, configure templates>> for more information.
|
| `request.template` | no | - | The body of the search template. See <<templates, configure templates>> for more information.
|
||||||
|
@ -29,7 +29,6 @@ example, to get the message field from the first hit, use `ctx.payload.hits.hits
|
||||||
| `extract` | no | - | A array of JSON keys to extract from the search response and load as the payload. When a search generates a large response, you can use `extract` to select the relevant fields instead of loading the entire response.
|
| `extract` | no | - | A array of JSON keys to extract from the search response and load as the payload. When a search generates a large response, you can use `extract` to select the relevant fields instead of loading the entire response.
|
||||||
| `timeout` | no | 30s | The timeout for waiting for the search api call to return. If no response is returned within this time, the search input times out and fails.
|
| `timeout` | no | 30s | The timeout for waiting for the search api call to return. If no response is returned within this time, the search input times out and fails.
|
||||||
This setting overrides the default internal search operations <<default-internal-ops-timeouts, timeouts>>.
|
This setting overrides the default internal search operations <<default-internal-ops-timeouts, timeouts>>.
|
||||||
| `dynamic_name_timezone` | no | - | The time zone to use for resolving the index name based on <<dynamic-index-names, Dynamic Index Names>>. The default time zone also can be <<dynamic-index-name-timezone, configured>> globally.
|
|
||||||
|======
|
|======
|
||||||
|
|
||||||
You can reference the following variables in the execution context when specifying the request `body`:
|
You can reference the following variables in the execution context when specifying the request `body`:
|
||||||
|
|
|
@ -56,7 +56,7 @@ The following table lists all available settings for the search transform:
|
||||||
|======
|
|======
|
||||||
| Name |Required | Default | Description
|
| Name |Required | Default | Description
|
||||||
| `request.search_type` | no | {ref}/search-request-search-type.html#query-then-fetch[query_then_fetch] | The search {ref}/search-request-search-type.html[search type]
|
| `request.search_type` | no | {ref}/search-request-search-type.html#query-then-fetch[query_then_fetch] | The search {ref}/search-request-search-type.html[search type]
|
||||||
| `request.indices` | no | all indices | One or more indices to search on (may be a comma-delimited string or an array of indices names). <<dynamic-index-names, Dynamic index names>> are supported.
|
| `request.indices` | no | all indices | One or more indices to search on (may be a comma-delimited string or an array of indices names).
|
||||||
| `request.types` | no | all types | One or more document types to search on (may be a comma-delimited string or an array of document types names)
|
| `request.types` | no | all types | One or more document types to search on (may be a comma-delimited string or an array of document types names)
|
||||||
| `request.body` | no | `match_all` query | The body of the request. The {ref}/search-request-body.html[request body] follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` <<templates, templates>>.
|
| `request.body` | no | `match_all` query | The body of the request. The {ref}/search-request-body.html[request body] follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` <<templates, templates>>.
|
||||||
| `request.indices_options.expand_wildcards` | no | `open` | Determines how to expand indices wildcards. Can be one of `open`, `closed`, `none` or `all` (see {ref}/multi-index.html[multi-index support])
|
| `request.indices_options.expand_wildcards` | no | `open` | Determines how to expand indices wildcards. Can be one of `open`, `closed`, `none` or `all` (see {ref}/multi-index.html[multi-index support])
|
||||||
|
@ -65,7 +65,6 @@ The following table lists all available settings for the search transform:
|
||||||
| `request.template` | no | - | The body of the search template. See <<templates, configure templates>> for more information.
|
| `request.template` | no | - | The body of the search template. See <<templates, configure templates>> for more information.
|
||||||
| `timeout` | no | 30s | The timeout for waiting for the search api call to return. If no response is returned within this time, the search transform times out and fails.
|
| `timeout` | no | 30s | The timeout for waiting for the search api call to return. If no response is returned within this time, the search transform times out and fails.
|
||||||
This setting overrides the default internal search operations <<default-internal-ops-timeouts, timeouts>>.
|
This setting overrides the default internal search operations <<default-internal-ops-timeouts, timeouts>>.
|
||||||
| `dynamic_name_timezone` | no | - | The time zone to use for resolving the index name based on <<dynamic-index-names, Dynamic Index Names>>. The default time zone also can be <<dynamic-index-name-timezone, configured>> globally.
|
|
||||||
|======
|
|======
|
||||||
|
|
||||||
[[transform-search-template]]
|
[[transform-search-template]]
|
||||||
|
|
|
@ -36,6 +36,21 @@ bin/plugin remove watcher
|
||||||
[[change-list]]
|
[[change-list]]
|
||||||
=== Change List
|
=== Change List
|
||||||
|
|
||||||
|
[float]
|
||||||
|
==== 2.0.0-rc1
|
||||||
|
|
||||||
|
.Breaking
|
||||||
|
* The dynamic index names support has been removed and Elasticsearch's date math index names support should be used instead.
|
||||||
|
The only difference between Watcher's dynamic index names support and Elasticsearch's date math index names support is
|
||||||
|
how timezones are expressed. In Watcher this is done via node settings, in Elasticsearch the timezone is part of the
|
||||||
|
date math index names support. Only if you're using dynamic index names with timezones in Watcher then you need to
|
||||||
|
upgrade your watches after the upgrade, otherwise your watches will work as they did before the upgrade. For example if
|
||||||
|
`watcher.dynamic_indices.time_zone` setting was set to `+01:00` and a watch has the following index name `<logstash-{now/d}>`
|
||||||
|
then after the upgrade you need to update this watch to use the following index name `<logstash-{now/d{YYYY.MM.dd|+01:00}}>`.
|
||||||
|
|
||||||
|
.Bug Fixes
|
||||||
|
* Fixed url encoding issue in http input and webhook output. The url params were url encoded twice.
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
==== 2.0.0-beta2
|
==== 2.0.0-beta2
|
||||||
|
|
||||||
|
@ -73,7 +88,7 @@ been aligned with the Elasticsearch versioning.
|
||||||
==== 1.0.0
|
==== 1.0.0
|
||||||
|
|
||||||
.Enhancements
|
.Enhancements
|
||||||
* Added execution time aware <<dynamic-index-names, dynamic index names>> support to `index`
|
* Added execution time aware dynamic index names support to `index`
|
||||||
action, `search` input, and `search` transform.
|
action, `search` input, and `search` transform.
|
||||||
* You must now explicitly specify the unit when configuring any time value. (Numeric-only
|
* You must now explicitly specify the unit when configuring any time value. (Numeric-only
|
||||||
values are no longer supported.)
|
values are no longer supported.)
|
||||||
|
|
|
@ -20,7 +20,6 @@ import org.elasticsearch.watcher.actions.Action;
|
||||||
import org.elasticsearch.watcher.actions.ExecutableAction;
|
import org.elasticsearch.watcher.actions.ExecutableAction;
|
||||||
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
||||||
import org.elasticsearch.watcher.support.ArrayObjectIterator;
|
import org.elasticsearch.watcher.support.ArrayObjectIterator;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.WatcherDateTimeUtils;
|
import org.elasticsearch.watcher.support.WatcherDateTimeUtils;
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
import org.elasticsearch.watcher.support.xcontent.XContentSource;
|
import org.elasticsearch.watcher.support.xcontent.XContentSource;
|
||||||
|
@ -37,17 +36,11 @@ public class ExecutableIndexAction extends ExecutableAction<IndexAction> {
|
||||||
|
|
||||||
private final ClientProxy client;
|
private final ClientProxy client;
|
||||||
private final TimeValue timeout;
|
private final TimeValue timeout;
|
||||||
private final DynamicIndexName indexName;
|
|
||||||
|
|
||||||
public ExecutableIndexAction(IndexAction action, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout, DynamicIndexName.Parser indexNameParser) {
|
public ExecutableIndexAction(IndexAction action, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout) {
|
||||||
super(action, logger);
|
super(action, logger);
|
||||||
this.client = client;
|
this.client = client;
|
||||||
this.timeout = action.timeout != null ? action.timeout : defaultTimeout;
|
this.timeout = action.timeout != null ? action.timeout : defaultTimeout;
|
||||||
this.indexName = indexNameParser.parse(action.index, action.dynamicNameTimeZone);
|
|
||||||
}
|
|
||||||
|
|
||||||
DynamicIndexName indexName() {
|
|
||||||
return indexName;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
|
@ -71,7 +64,7 @@ public class ExecutableIndexAction extends ExecutableAction<IndexAction> {
|
||||||
|
|
||||||
IndexRequest indexRequest = new IndexRequest();
|
IndexRequest indexRequest = new IndexRequest();
|
||||||
|
|
||||||
indexRequest.index(indexName.name(ctx.executionTime()));
|
indexRequest.index(action.index);
|
||||||
indexRequest.type(action.docType);
|
indexRequest.type(action.docType);
|
||||||
|
|
||||||
if (action.executionTimeField != null && !TimestampFieldMapper.NAME.equals(action.executionTimeField)) {
|
if (action.executionTimeField != null && !TimestampFieldMapper.NAME.equals(action.executionTimeField)) {
|
||||||
|
@ -104,7 +97,7 @@ public class ExecutableIndexAction extends ExecutableAction<IndexAction> {
|
||||||
}
|
}
|
||||||
Map<String, Object> doc = (Map<String, Object>) item;
|
Map<String, Object> doc = (Map<String, Object>) item;
|
||||||
IndexRequest indexRequest = new IndexRequest();
|
IndexRequest indexRequest = new IndexRequest();
|
||||||
indexRequest.index(indexName.name(ctx.executionTime()));
|
indexRequest.index(action.index);
|
||||||
indexRequest.type(action.docType);
|
indexRequest.type(action.docType);
|
||||||
if (action.executionTimeField != null && !TimestampFieldMapper.NAME.equals(action.executionTimeField)) {
|
if (action.executionTimeField != null && !TimestampFieldMapper.NAME.equals(action.executionTimeField)) {
|
||||||
if (!(doc instanceof HashMap)) {
|
if (!(doc instanceof HashMap)) {
|
||||||
|
|
|
@ -12,7 +12,6 @@ import org.elasticsearch.common.unit.TimeValue;
|
||||||
import org.elasticsearch.common.xcontent.XContentParser;
|
import org.elasticsearch.common.xcontent.XContentParser;
|
||||||
import org.elasticsearch.watcher.actions.ActionFactory;
|
import org.elasticsearch.watcher.actions.ActionFactory;
|
||||||
import org.elasticsearch.watcher.actions.email.ExecutableEmailAction;
|
import org.elasticsearch.watcher.actions.email.ExecutableEmailAction;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
|
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
|
@ -23,14 +22,12 @@ import java.io.IOException;
|
||||||
public class IndexActionFactory extends ActionFactory<IndexAction, ExecutableIndexAction> {
|
public class IndexActionFactory extends ActionFactory<IndexAction, ExecutableIndexAction> {
|
||||||
|
|
||||||
private final ClientProxy client;
|
private final ClientProxy client;
|
||||||
private final DynamicIndexName.Parser indexNamesParser;
|
|
||||||
private final TimeValue defaultTimeout;
|
private final TimeValue defaultTimeout;
|
||||||
|
|
||||||
@Inject
|
@Inject
|
||||||
public IndexActionFactory(Settings settings, ClientProxy client) {
|
public IndexActionFactory(Settings settings, ClientProxy client) {
|
||||||
super(Loggers.getLogger(ExecutableEmailAction.class, settings));
|
super(Loggers.getLogger(ExecutableEmailAction.class, settings));
|
||||||
this.client = client;
|
this.client = client;
|
||||||
this.indexNamesParser = new DynamicIndexName.Parser(settings, "watcher.actions.index");
|
|
||||||
this.defaultTimeout = settings.getAsTime("watcher.actions.index.default_timeout", null);
|
this.defaultTimeout = settings.getAsTime("watcher.actions.index.default_timeout", null);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -46,6 +43,6 @@ public class IndexActionFactory extends ActionFactory<IndexAction, ExecutableInd
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public ExecutableIndexAction createExecutable(IndexAction action) {
|
public ExecutableIndexAction createExecutable(IndexAction action) {
|
||||||
return new ExecutableIndexAction(action, actionLogger, client, defaultTimeout, indexNamesParser);
|
return new ExecutableIndexAction(action, actionLogger, client, defaultTimeout);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
@ -18,7 +18,6 @@ import org.elasticsearch.common.xcontent.XContentParser;
|
||||||
import org.elasticsearch.search.SearchHit;
|
import org.elasticsearch.search.SearchHit;
|
||||||
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
||||||
import org.elasticsearch.watcher.input.ExecutableInput;
|
import org.elasticsearch.watcher.input.ExecutableInput;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.WatcherUtils;
|
import org.elasticsearch.watcher.support.WatcherUtils;
|
||||||
import org.elasticsearch.watcher.support.XContentFilterKeysUtils;
|
import org.elasticsearch.watcher.support.XContentFilterKeysUtils;
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
|
@ -37,24 +36,17 @@ public class ExecutableSearchInput extends ExecutableInput<SearchInput, SearchIn
|
||||||
|
|
||||||
private final ClientProxy client;
|
private final ClientProxy client;
|
||||||
private final @Nullable TimeValue timeout;
|
private final @Nullable TimeValue timeout;
|
||||||
private final @Nullable DynamicIndexName[] indexNames;
|
|
||||||
|
|
||||||
public ExecutableSearchInput(SearchInput input, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout, DynamicIndexName.Parser indexNameParser) {
|
public ExecutableSearchInput(SearchInput input, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout) {
|
||||||
super(input, logger);
|
super(input, logger);
|
||||||
this.client = client;
|
this.client = client;
|
||||||
this.timeout = input.getTimeout() != null ? input.getTimeout() : defaultTimeout;
|
this.timeout = input.getTimeout() != null ? input.getTimeout() : defaultTimeout;
|
||||||
String[] indices = input.getSearchRequest().indices();
|
|
||||||
indexNames = indices != null ? indexNameParser.parse(indices, input.getDynamicNameTimeZone()) : null;
|
|
||||||
}
|
|
||||||
|
|
||||||
DynamicIndexName[] indexNames() {
|
|
||||||
return indexNames;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public SearchInput.Result execute(WatchExecutionContext ctx) {
|
public SearchInput.Result execute(WatchExecutionContext ctx) {
|
||||||
SearchRequest request = null;
|
SearchRequest request = null;
|
||||||
try {
|
try {
|
||||||
request = WatcherUtils.createSearchRequestFromPrototype(input.getSearchRequest(), indexNames, ctx, null);
|
request = WatcherUtils.createSearchRequestFromPrototype(input.getSearchRequest(), ctx, null);
|
||||||
return doExecute(ctx, request);
|
return doExecute(ctx, request);
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
logger.error("failed to execute [{}] input for [{}]", e, SearchInput.TYPE, ctx.watch());
|
logger.error("failed to execute [{}] input for [{}]", e, SearchInput.TYPE, ctx.watch());
|
||||||
|
|
|
@ -12,7 +12,6 @@ import org.elasticsearch.common.unit.TimeValue;
|
||||||
import org.elasticsearch.common.xcontent.XContentParser;
|
import org.elasticsearch.common.xcontent.XContentParser;
|
||||||
import org.elasticsearch.watcher.input.InputFactory;
|
import org.elasticsearch.watcher.input.InputFactory;
|
||||||
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
|
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
|
@ -23,14 +22,12 @@ import java.io.IOException;
|
||||||
public class SearchInputFactory extends InputFactory<SearchInput, SearchInput.Result, ExecutableSearchInput> {
|
public class SearchInputFactory extends InputFactory<SearchInput, SearchInput.Result, ExecutableSearchInput> {
|
||||||
|
|
||||||
private final ClientProxy client;
|
private final ClientProxy client;
|
||||||
private final DynamicIndexName.Parser indexNameParser;
|
|
||||||
private final TimeValue defaultTimeout;
|
private final TimeValue defaultTimeout;
|
||||||
|
|
||||||
@Inject
|
@Inject
|
||||||
public SearchInputFactory(Settings settings, ClientProxy client) {
|
public SearchInputFactory(Settings settings, ClientProxy client) {
|
||||||
super(Loggers.getLogger(ExecutableSimpleInput.class, settings));
|
super(Loggers.getLogger(ExecutableSimpleInput.class, settings));
|
||||||
this.client = client;
|
this.client = client;
|
||||||
this.indexNameParser = new DynamicIndexName.Parser(settings, "watcher.input.search");
|
|
||||||
this.defaultTimeout = settings.getAsTime("watcher.input.search.default_timeout", null);
|
this.defaultTimeout = settings.getAsTime("watcher.input.search.default_timeout", null);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -46,6 +43,6 @@ public class SearchInputFactory extends InputFactory<SearchInput, SearchInput.Re
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public ExecutableSearchInput createExecutable(SearchInput input) {
|
public ExecutableSearchInput createExecutable(SearchInput input) {
|
||||||
return new ExecutableSearchInput(input, inputLogger, client, defaultTimeout, indexNameParser);
|
return new ExecutableSearchInput(input, inputLogger, client, defaultTimeout);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
@ -1,339 +0,0 @@
|
||||||
/*
|
|
||||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
|
||||||
* or more contributor license agreements. Licensed under the Elastic License;
|
|
||||||
* you may not use this file except in compliance with the Elastic License.
|
|
||||||
*/
|
|
||||||
package org.elasticsearch.watcher.support;
|
|
||||||
|
|
||||||
import org.elasticsearch.ElasticsearchParseException;
|
|
||||||
import org.elasticsearch.common.Nullable;
|
|
||||||
import org.elasticsearch.common.joda.DateMathParser;
|
|
||||||
import org.elasticsearch.common.joda.FormatDateTimeFormatter;
|
|
||||||
import org.elasticsearch.common.settings.Settings;
|
|
||||||
import org.elasticsearch.common.xcontent.ToXContent;
|
|
||||||
import org.elasticsearch.common.xcontent.XContentBuilder;
|
|
||||||
import org.joda.time.DateTime;
|
|
||||||
import org.joda.time.DateTimeZone;
|
|
||||||
import org.joda.time.format.DateTimeFormat;
|
|
||||||
import org.joda.time.format.DateTimeFormatter;
|
|
||||||
|
|
||||||
import java.io.IOException;
|
|
||||||
import java.util.ArrayList;
|
|
||||||
import java.util.List;
|
|
||||||
import java.util.Locale;
|
|
||||||
import java.util.concurrent.Callable;
|
|
||||||
|
|
||||||
/**
|
|
||||||
*
|
|
||||||
*/
|
|
||||||
public class DynamicIndexName implements ToXContent {
|
|
||||||
|
|
||||||
public static final String DEFAULT_DATE_FORMAT = "YYYY.MM.dd";
|
|
||||||
|
|
||||||
private static final String EXPRESSION_LEFT_BOUND = "<";
|
|
||||||
private static final String EXPRESSION_RIGHT_BOUND = ">";
|
|
||||||
private static final char LEFT_BOUND = '{';
|
|
||||||
private static final char RIGHT_BOUND = '}';
|
|
||||||
private static final char ESCAPE_CHAR = '\\';
|
|
||||||
|
|
||||||
private final String text;
|
|
||||||
private final Expression expression;
|
|
||||||
|
|
||||||
private DynamicIndexName(String text, Expression expression) {
|
|
||||||
this.text = text;
|
|
||||||
this.expression = expression;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String text() {
|
|
||||||
return text;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String name(DateTime now) {
|
|
||||||
return expression.eval(now);
|
|
||||||
}
|
|
||||||
|
|
||||||
public static String[] names(DynamicIndexName[] indexNames, DateTime now) {
|
|
||||||
String[] names = new String[indexNames.length];
|
|
||||||
for (int i = 0; i < names.length; i++) {
|
|
||||||
names[i] = indexNames[i].name(now);
|
|
||||||
}
|
|
||||||
return names;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Override
|
|
||||||
public boolean equals(Object o) {
|
|
||||||
if (this == o) return true;
|
|
||||||
if (o == null || getClass() != o.getClass()) return false;
|
|
||||||
|
|
||||||
DynamicIndexName that = (DynamicIndexName) o;
|
|
||||||
|
|
||||||
return text.equals(that.text);
|
|
||||||
}
|
|
||||||
|
|
||||||
@Override
|
|
||||||
public int hashCode() {
|
|
||||||
return text.hashCode();
|
|
||||||
}
|
|
||||||
|
|
||||||
@Override
|
|
||||||
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
|
|
||||||
return builder.value(text);
|
|
||||||
}
|
|
||||||
|
|
||||||
public static String defaultDateFormat(Settings settings, String componentPrefix) {
|
|
||||||
if (componentPrefix == null) {
|
|
||||||
return defaultDateFormat(settings);
|
|
||||||
}
|
|
||||||
return settings.get(componentPrefix + ".dynamic_indices.default_date_format", defaultDateFormat(settings));
|
|
||||||
}
|
|
||||||
|
|
||||||
public static String defaultDateFormat(Settings settings) {
|
|
||||||
return settings.get("watcher.dynamic_indices.default_date_format", DEFAULT_DATE_FORMAT);
|
|
||||||
}
|
|
||||||
|
|
||||||
public static DateTimeZone timeZone(Settings settings, String componentPrefix) {
|
|
||||||
if (componentPrefix == null) {
|
|
||||||
return timeZone(settings);
|
|
||||||
}
|
|
||||||
|
|
||||||
String timeZoneId = settings.get(componentPrefix + ".dynamic_indices.time_zone", DateTimeZone.UTC.getID());
|
|
||||||
return DateTimeZone.forID(timeZoneId);
|
|
||||||
}
|
|
||||||
|
|
||||||
public static DateTimeZone timeZone(Settings settings) {
|
|
||||||
String timeZoneId = settings.get("watcher.dynamic_indices.time_zone", DateTimeZone.UTC.getID());
|
|
||||||
return DateTimeZone.forID(timeZoneId);
|
|
||||||
}
|
|
||||||
|
|
||||||
interface Expression {
|
|
||||||
|
|
||||||
String eval(DateTime now);
|
|
||||||
|
|
||||||
}
|
|
||||||
|
|
||||||
static class StaticExpression implements Expression {
|
|
||||||
|
|
||||||
private final String value;
|
|
||||||
|
|
||||||
public StaticExpression(String value) {
|
|
||||||
this.value = value;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Override
|
|
||||||
public String eval(DateTime now) {
|
|
||||||
return value;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
static class DateMathExpression implements Expression {
|
|
||||||
|
|
||||||
private final DateMathParser dateMathParser;
|
|
||||||
private final String mathExpression;
|
|
||||||
private final FormatDateTimeFormatter formatter;
|
|
||||||
private final DateTimeZone timeZone;
|
|
||||||
|
|
||||||
public DateMathExpression(String defaultFormat, DateTimeZone timeZone, String expression) {
|
|
||||||
this.timeZone = timeZone;
|
|
||||||
int i = expression.indexOf(LEFT_BOUND);
|
|
||||||
String format;
|
|
||||||
if (i < 0) {
|
|
||||||
mathExpression = expression;
|
|
||||||
format = defaultFormat;
|
|
||||||
} else {
|
|
||||||
if (expression.lastIndexOf(RIGHT_BOUND) != expression.length() - 1) {
|
|
||||||
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. missing closing `}` for date math format", expression);
|
|
||||||
}
|
|
||||||
if (i == expression.length() - 2) {
|
|
||||||
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. missing date format", expression);
|
|
||||||
}
|
|
||||||
mathExpression = expression.substring(0, i);
|
|
||||||
format = expression.substring(i + 1, expression.length() - 1);
|
|
||||||
|
|
||||||
}
|
|
||||||
DateTimeFormatter parser = DateTimeFormat.forPattern(format).withZone(timeZone);
|
|
||||||
formatter = new FormatDateTimeFormatter(defaultFormat, parser, Locale.ROOT);
|
|
||||||
dateMathParser = new DateMathParser(formatter);
|
|
||||||
}
|
|
||||||
|
|
||||||
@Override
|
|
||||||
public String eval(final DateTime now) {
|
|
||||||
long millis = dateMathParser.parse(mathExpression, new Callable<Long>() {
|
|
||||||
@Override
|
|
||||||
public Long call() throws Exception {
|
|
||||||
return now.getMillis();
|
|
||||||
}
|
|
||||||
}, false, timeZone);
|
|
||||||
return formatter.printer().print(millis);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
static class CompoundExpression implements Expression {
|
|
||||||
|
|
||||||
private final Expression[] parts;
|
|
||||||
|
|
||||||
public CompoundExpression(Expression[] parts) {
|
|
||||||
this.parts = parts;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Override
|
|
||||||
public String eval(DateTime now) {
|
|
||||||
StringBuilder sb = new StringBuilder();
|
|
||||||
for (Expression part : parts) {
|
|
||||||
sb.append(part.eval(now));
|
|
||||||
}
|
|
||||||
return sb.toString();
|
|
||||||
}
|
|
||||||
|
|
||||||
static Expression parse(String defaultDateFormat, DateTimeZone timeZone, char[] text, int from, int length) {
|
|
||||||
boolean dynamic = false;
|
|
||||||
List<Expression> expressions = new ArrayList<>();
|
|
||||||
StringBuilder sb = new StringBuilder();
|
|
||||||
boolean inPlaceHolder = false;
|
|
||||||
boolean inDateFormat = false;
|
|
||||||
boolean escape = false;
|
|
||||||
for (int i = from; i < length; i++) {
|
|
||||||
boolean escapedChar = escape;
|
|
||||||
if (escape) {
|
|
||||||
escape = false;
|
|
||||||
}
|
|
||||||
char c = text[i];
|
|
||||||
|
|
||||||
if (c == ESCAPE_CHAR) {
|
|
||||||
if (escapedChar) {
|
|
||||||
sb.append(c);
|
|
||||||
escape = false;
|
|
||||||
} else {
|
|
||||||
escape = true;
|
|
||||||
}
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (inPlaceHolder) {
|
|
||||||
switch (c) {
|
|
||||||
|
|
||||||
case LEFT_BOUND:
|
|
||||||
if (inDateFormat && escapedChar) {
|
|
||||||
sb.append(c);
|
|
||||||
} else if (!inDateFormat) {
|
|
||||||
inDateFormat = true;
|
|
||||||
sb.append(c);
|
|
||||||
} else {
|
|
||||||
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. invalid character in placeholder at position [{}]", new String(text, from, length), i);
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
|
|
||||||
case RIGHT_BOUND:
|
|
||||||
if (inDateFormat && escapedChar) {
|
|
||||||
sb.append(c);
|
|
||||||
} else if (inDateFormat) {
|
|
||||||
inDateFormat = false;
|
|
||||||
sb.append(c);
|
|
||||||
} else {
|
|
||||||
expressions.add(new DateMathExpression(defaultDateFormat, timeZone, sb.toString()));
|
|
||||||
sb = new StringBuilder();
|
|
||||||
inPlaceHolder = false;
|
|
||||||
dynamic = true;
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
|
|
||||||
default:
|
|
||||||
sb.append(c);
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
switch (c) {
|
|
||||||
|
|
||||||
case LEFT_BOUND:
|
|
||||||
if (escapedChar) {
|
|
||||||
sb.append(c);
|
|
||||||
} else {
|
|
||||||
expressions.add(new StaticExpression(sb.toString()));
|
|
||||||
sb = new StringBuilder();
|
|
||||||
inPlaceHolder = true;
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
|
|
||||||
case RIGHT_BOUND:
|
|
||||||
if (!escapedChar) {
|
|
||||||
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. invalid character at position [{}]. " +
|
|
||||||
"`{` and `}` are reserved characters and should be escaped when used as part of the index name using `\\` (e.g. `\\{text\\}`)", new String(text, from, length), i);
|
|
||||||
}
|
|
||||||
sb.append(c);
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
sb.append(c);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (inPlaceHolder) {
|
|
||||||
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. date math placeholder is open ended", new String(text, from, length));
|
|
||||||
}
|
|
||||||
if (sb.length() > 0) {
|
|
||||||
expressions.add(new StaticExpression(sb.toString()));
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!dynamic) {
|
|
||||||
// if all the expressions are static... lets optimize to a single static expression
|
|
||||||
sb = new StringBuilder();
|
|
||||||
for (Expression expression : expressions) {
|
|
||||||
sb.append(((StaticExpression) expression).value);
|
|
||||||
}
|
|
||||||
return new StaticExpression(sb.toString());
|
|
||||||
}
|
|
||||||
|
|
||||||
if (expressions.size() == 1) {
|
|
||||||
return expressions.get(0);
|
|
||||||
}
|
|
||||||
|
|
||||||
return new CompoundExpression(expressions.toArray(new Expression[expressions.size()]));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public static class Parser {
|
|
||||||
|
|
||||||
private final String defaultDateFormat;
|
|
||||||
private final DateTimeZone timeZone;
|
|
||||||
|
|
||||||
public Parser() {
|
|
||||||
this(DEFAULT_DATE_FORMAT, DateTimeZone.UTC);
|
|
||||||
}
|
|
||||||
|
|
||||||
public Parser(String defaultDateFormat, DateTimeZone timeZone) {
|
|
||||||
this.defaultDateFormat = defaultDateFormat;
|
|
||||||
this.timeZone = timeZone;
|
|
||||||
}
|
|
||||||
|
|
||||||
public Parser(Settings settings, String componentPrefix) {
|
|
||||||
this(defaultDateFormat(settings, componentPrefix), timeZone(settings, componentPrefix));
|
|
||||||
}
|
|
||||||
|
|
||||||
public DynamicIndexName parse(String template) {
|
|
||||||
return parse(template, null);
|
|
||||||
}
|
|
||||||
|
|
||||||
public DynamicIndexName parse(String template, @Nullable DateTimeZone timeZone) {
|
|
||||||
DateTimeZone tz = timeZone != null ? timeZone : this.timeZone;
|
|
||||||
if (template == null) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
if (!template.startsWith(EXPRESSION_LEFT_BOUND) || !template.endsWith(EXPRESSION_RIGHT_BOUND)) {
|
|
||||||
return new DynamicIndexName(template, new StaticExpression(template));
|
|
||||||
}
|
|
||||||
return new DynamicIndexName(template, CompoundExpression.parse(defaultDateFormat, tz, template.toCharArray(), 1, template.length() - 1));
|
|
||||||
}
|
|
||||||
|
|
||||||
public DynamicIndexName[] parse(String[] templates) {
|
|
||||||
return parse(templates, null);
|
|
||||||
}
|
|
||||||
|
|
||||||
public DynamicIndexName[] parse(String[] templates, @Nullable DateTimeZone timeZone) {
|
|
||||||
if (templates.length == 0) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
DynamicIndexName[] dynamicIndexNames = new DynamicIndexName[templates.length];
|
|
||||||
for (int i = 0; i < dynamicIndexNames.length; i++) {
|
|
||||||
dynamicIndexNames[i] = parse(templates[i], timeZone);
|
|
||||||
}
|
|
||||||
return dynamicIndexNames;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
|
@ -53,16 +53,11 @@ public final class WatcherUtils {
|
||||||
return XContentHelper.convertToMap(builder.bytes(), false).v2();
|
return XContentHelper.convertToMap(builder.bytes(), false).v2();
|
||||||
}
|
}
|
||||||
|
|
||||||
public static SearchRequest createSearchRequestFromPrototype(SearchRequest requestPrototype, @Nullable DynamicIndexName[] dynamicIndexNames, WatchExecutionContext ctx, Payload payload) throws IOException {
|
public static SearchRequest createSearchRequestFromPrototype(SearchRequest requestPrototype, WatchExecutionContext ctx, Payload payload) throws IOException {
|
||||||
|
|
||||||
String[] indices = dynamicIndexNames == null ?
|
|
||||||
requestPrototype.indices() :
|
|
||||||
DynamicIndexName.names(dynamicIndexNames, ctx.executionTime());
|
|
||||||
|
|
||||||
SearchRequest request = new SearchRequest(requestPrototype)
|
SearchRequest request = new SearchRequest(requestPrototype)
|
||||||
.indicesOptions(requestPrototype.indicesOptions())
|
.indicesOptions(requestPrototype.indicesOptions())
|
||||||
.searchType(requestPrototype.searchType())
|
.searchType(requestPrototype.searchType())
|
||||||
.indices(indices)
|
.indices(requestPrototype.indices())
|
||||||
.types(requestPrototype.types());
|
.types(requestPrototype.types());
|
||||||
|
|
||||||
// TODO: Revise this search template conversion code once search templates in core have been refactored once ES 2.0 is released.
|
// TODO: Revise this search template conversion code once search templates in core have been refactored once ES 2.0 is released.
|
||||||
|
|
|
@ -12,7 +12,6 @@ import org.elasticsearch.common.Nullable;
|
||||||
import org.elasticsearch.common.logging.ESLogger;
|
import org.elasticsearch.common.logging.ESLogger;
|
||||||
import org.elasticsearch.common.unit.TimeValue;
|
import org.elasticsearch.common.unit.TimeValue;
|
||||||
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.WatcherUtils;
|
import org.elasticsearch.watcher.support.WatcherUtils;
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
import org.elasticsearch.watcher.transform.ExecutableTransform;
|
import org.elasticsearch.watcher.transform.ExecutableTransform;
|
||||||
|
@ -27,25 +26,18 @@ public class ExecutableSearchTransform extends ExecutableTransform<SearchTransfo
|
||||||
|
|
||||||
protected final ClientProxy client;
|
protected final ClientProxy client;
|
||||||
protected final @Nullable TimeValue timeout;
|
protected final @Nullable TimeValue timeout;
|
||||||
private final @Nullable DynamicIndexName[] indexNames;
|
|
||||||
|
|
||||||
public ExecutableSearchTransform(SearchTransform transform, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout, DynamicIndexName.Parser indexNameParser) {
|
public ExecutableSearchTransform(SearchTransform transform, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout) {
|
||||||
super(transform, logger);
|
super(transform, logger);
|
||||||
this.client = client;
|
this.client = client;
|
||||||
this.timeout = transform.getTimeout() != null ? transform.getTimeout() : defaultTimeout;
|
this.timeout = transform.getTimeout() != null ? transform.getTimeout() : defaultTimeout;
|
||||||
String[] indices = transform.getRequest().indices();
|
|
||||||
this.indexNames = indices != null ? indexNameParser.parse(indices, transform.getDynamicNameTimeZone()) : null;
|
|
||||||
}
|
|
||||||
|
|
||||||
DynamicIndexName[] indexNames() {
|
|
||||||
return indexNames;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public SearchTransform.Result execute(WatchExecutionContext ctx, Payload payload) {
|
public SearchTransform.Result execute(WatchExecutionContext ctx, Payload payload) {
|
||||||
SearchRequest request = null;
|
SearchRequest request = null;
|
||||||
try {
|
try {
|
||||||
request = WatcherUtils.createSearchRequestFromPrototype(transform.getRequest(), indexNames, ctx, payload);
|
request = WatcherUtils.createSearchRequestFromPrototype(transform.getRequest(), ctx, payload);
|
||||||
SearchResponse resp = client.search(request, timeout);
|
SearchResponse resp = client.search(request, timeout);
|
||||||
return new SearchTransform.Result(request, new Payload.XContent(resp));
|
return new SearchTransform.Result(request, new Payload.XContent(resp));
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
|
|
|
@ -10,7 +10,6 @@ import org.elasticsearch.common.logging.Loggers;
|
||||||
import org.elasticsearch.common.settings.Settings;
|
import org.elasticsearch.common.settings.Settings;
|
||||||
import org.elasticsearch.common.unit.TimeValue;
|
import org.elasticsearch.common.unit.TimeValue;
|
||||||
import org.elasticsearch.common.xcontent.XContentParser;
|
import org.elasticsearch.common.xcontent.XContentParser;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
import org.elasticsearch.watcher.transform.TransformFactory;
|
import org.elasticsearch.watcher.transform.TransformFactory;
|
||||||
|
|
||||||
|
@ -22,14 +21,12 @@ import java.io.IOException;
|
||||||
public class SearchTransformFactory extends TransformFactory<SearchTransform, SearchTransform.Result, ExecutableSearchTransform> {
|
public class SearchTransformFactory extends TransformFactory<SearchTransform, SearchTransform.Result, ExecutableSearchTransform> {
|
||||||
|
|
||||||
protected final ClientProxy client;
|
protected final ClientProxy client;
|
||||||
protected final DynamicIndexName.Parser indexNameParser;
|
|
||||||
private final TimeValue defaultTimeout;
|
private final TimeValue defaultTimeout;
|
||||||
|
|
||||||
@Inject
|
@Inject
|
||||||
public SearchTransformFactory(Settings settings, ClientProxy client) {
|
public SearchTransformFactory(Settings settings, ClientProxy client) {
|
||||||
super(Loggers.getLogger(ExecutableSearchTransform.class, settings));
|
super(Loggers.getLogger(ExecutableSearchTransform.class, settings));
|
||||||
this.client = client;
|
this.client = client;
|
||||||
this.indexNameParser = new DynamicIndexName.Parser(settings, "watcher.transform.search");
|
|
||||||
this.defaultTimeout = settings.getAsTime("watcher.transform.search.default_timeout", null);
|
this.defaultTimeout = settings.getAsTime("watcher.transform.search.default_timeout", null);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -45,6 +42,6 @@ public class SearchTransformFactory extends TransformFactory<SearchTransform, Se
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public ExecutableSearchTransform createExecutable(SearchTransform transform) {
|
public ExecutableSearchTransform createExecutable(SearchTransform transform) {
|
||||||
return new ExecutableSearchTransform(transform, transformLogger, client, defaultTimeout, indexNameParser);
|
return new ExecutableSearchTransform(transform, transformLogger, client, defaultTimeout);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
@ -21,15 +21,12 @@ import org.elasticsearch.test.ESIntegTestCase;
|
||||||
import org.elasticsearch.watcher.actions.Action;
|
import org.elasticsearch.watcher.actions.Action;
|
||||||
import org.elasticsearch.watcher.actions.Action.Result.Status;
|
import org.elasticsearch.watcher.actions.Action.Result.Status;
|
||||||
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.WatcherDateTimeUtils;
|
import org.elasticsearch.watcher.support.WatcherDateTimeUtils;
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
import org.elasticsearch.watcher.support.xcontent.XContentSource;
|
import org.elasticsearch.watcher.support.xcontent.XContentSource;
|
||||||
import org.elasticsearch.watcher.test.WatcherTestUtils;
|
import org.elasticsearch.watcher.test.WatcherTestUtils;
|
||||||
import org.elasticsearch.watcher.watch.Payload;
|
import org.elasticsearch.watcher.watch.Payload;
|
||||||
import org.joda.time.DateTime;
|
import org.joda.time.DateTime;
|
||||||
import org.joda.time.DateTimeZone;
|
|
||||||
import org.joda.time.format.DateTimeFormat;
|
|
||||||
import org.junit.Test;
|
import org.junit.Test;
|
||||||
|
|
||||||
import java.util.Arrays;
|
import java.util.Arrays;
|
||||||
|
@ -73,7 +70,7 @@ public class IndexActionTests extends ESIntegTestCase {
|
||||||
}
|
}
|
||||||
|
|
||||||
IndexAction action = new IndexAction("test-index", "test-type", timestampField, null, null);
|
IndexAction action = new IndexAction("test-index", "test-type", timestampField, null, null);
|
||||||
ExecutableIndexAction executable = new ExecutableIndexAction(action, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableIndexAction executable = new ExecutableIndexAction(action, logger, ClientProxy.of(client()), null);
|
||||||
DateTime executionTime = DateTime.now(UTC);
|
DateTime executionTime = DateTime.now(UTC);
|
||||||
Payload payload = randomBoolean() ? new Payload.Simple("foo", "bar") : new Payload.Simple("_doc", ImmutableMap.of("foo", "bar"));
|
Payload payload = randomBoolean() ? new Payload.Simple("foo", "bar") : new Payload.Simple("_doc", ImmutableMap.of("foo", "bar"));
|
||||||
WatchExecutionContext ctx = WatcherTestUtils.mockExecutionContext("_id", executionTime, payload);
|
WatchExecutionContext ctx = WatcherTestUtils.mockExecutionContext("_id", executionTime, payload);
|
||||||
|
@ -136,7 +133,7 @@ public class IndexActionTests extends ESIntegTestCase {
|
||||||
);
|
);
|
||||||
|
|
||||||
IndexAction action = new IndexAction("test-index", "test-type", timestampField, null, null);
|
IndexAction action = new IndexAction("test-index", "test-type", timestampField, null, null);
|
||||||
ExecutableIndexAction executable = new ExecutableIndexAction(action, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableIndexAction executable = new ExecutableIndexAction(action, logger, ClientProxy.of(client()), null);
|
||||||
DateTime executionTime = DateTime.now(UTC);
|
DateTime executionTime = DateTime.now(UTC);
|
||||||
WatchExecutionContext ctx = WatcherTestUtils.mockExecutionContext("_id", executionTime, new Payload.Simple("_doc", list));
|
WatchExecutionContext ctx = WatcherTestUtils.mockExecutionContext("_id", executionTime, new Payload.Simple("_doc", list));
|
||||||
|
|
||||||
|
@ -217,47 +214,6 @@ public class IndexActionTests extends ESIntegTestCase {
|
||||||
assertThat(executable.action().timeout, equalTo(writeTimeout));
|
assertThat(executable.action().timeout, equalTo(writeTimeout));
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testParser_DynamicIndex() throws Exception {
|
|
||||||
|
|
||||||
DateTime now = DateTime.now(UTC);
|
|
||||||
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.forOffsetHours(-2) : null;
|
|
||||||
if (timeZone != null) {
|
|
||||||
now = now.withHourOfDay(0).withMinuteOfHour(0);
|
|
||||||
}
|
|
||||||
|
|
||||||
XContentBuilder builder = jsonBuilder();
|
|
||||||
builder.startObject()
|
|
||||||
.field(IndexAction.Field.INDEX.getPreferredName(), "<idx-{now/d}>")
|
|
||||||
.field(IndexAction.Field.DOC_TYPE.getPreferredName(), "test-type");
|
|
||||||
|
|
||||||
boolean timeZoneInWatch = randomBoolean();
|
|
||||||
if (timeZone != null && timeZoneInWatch) {
|
|
||||||
builder.field(IndexAction.Field.DYNAMIC_NAME_TIMEZONE.getPreferredName(), timeZone);
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.endObject();
|
|
||||||
|
|
||||||
Settings.Builder settings = Settings.builder();
|
|
||||||
if (timeZone != null && !timeZoneInWatch) {
|
|
||||||
settings.put("watcher.actions.index.dynamic_indices.time_zone", timeZone);
|
|
||||||
}
|
|
||||||
|
|
||||||
IndexActionFactory actionParser = new IndexActionFactory(settings.build(), ClientProxy.of(client()));
|
|
||||||
XContentParser parser = JsonXContent.jsonXContent.createParser(builder.bytes());
|
|
||||||
parser.nextToken();
|
|
||||||
|
|
||||||
ExecutableIndexAction executable = actionParser.parseExecutable(randomAsciiOfLength(5), randomAsciiOfLength(3), parser);
|
|
||||||
|
|
||||||
assertThat(executable, notNullValue());
|
|
||||||
assertThat(executable.action().index, is("<idx-{now/d}>"));
|
|
||||||
String indexName = executable.indexName().name(now);
|
|
||||||
if (timeZone != null) {
|
|
||||||
now = now.withZone(timeZone);
|
|
||||||
}
|
|
||||||
assertThat(indexName, is("idx-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now)));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
public void testParser_Failure() throws Exception {
|
public void testParser_Failure() throws Exception {
|
||||||
XContentBuilder builder = jsonBuilder();
|
XContentBuilder builder = jsonBuilder();
|
||||||
|
|
|
@ -7,7 +7,6 @@ package org.elasticsearch.watcher.input.search;
|
||||||
|
|
||||||
import com.google.common.collect.ImmutableMap;
|
import com.google.common.collect.ImmutableMap;
|
||||||
|
|
||||||
import org.elasticsearch.ElasticsearchParseException;
|
|
||||||
import org.elasticsearch.action.indexedscripts.put.PutIndexedScriptRequest;
|
import org.elasticsearch.action.indexedscripts.put.PutIndexedScriptRequest;
|
||||||
import org.elasticsearch.action.search.SearchRequest;
|
import org.elasticsearch.action.search.SearchRequest;
|
||||||
import org.elasticsearch.action.search.SearchType;
|
import org.elasticsearch.action.search.SearchType;
|
||||||
|
@ -20,7 +19,6 @@ import org.elasticsearch.common.xcontent.XContentParser;
|
||||||
import org.elasticsearch.common.xcontent.json.JsonXContent;
|
import org.elasticsearch.common.xcontent.json.JsonXContent;
|
||||||
import org.elasticsearch.common.xcontent.support.XContentMapValues;
|
import org.elasticsearch.common.xcontent.support.XContentMapValues;
|
||||||
import org.elasticsearch.search.builder.SearchSourceBuilder;
|
import org.elasticsearch.search.builder.SearchSourceBuilder;
|
||||||
import org.elasticsearch.search.sort.SortBuilders;
|
|
||||||
import org.elasticsearch.test.ESIntegTestCase;
|
import org.elasticsearch.test.ESIntegTestCase;
|
||||||
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
|
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
|
||||||
import org.elasticsearch.watcher.actions.ActionStatus;
|
import org.elasticsearch.watcher.actions.ActionStatus;
|
||||||
|
@ -31,7 +29,6 @@ import org.elasticsearch.watcher.execution.TriggeredExecutionContext;
|
||||||
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
||||||
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
||||||
import org.elasticsearch.watcher.input.simple.SimpleInput;
|
import org.elasticsearch.watcher.input.simple.SimpleInput;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
import org.elasticsearch.watcher.support.text.TextTemplate;
|
import org.elasticsearch.watcher.support.text.TextTemplate;
|
||||||
import org.elasticsearch.watcher.trigger.schedule.IntervalSchedule;
|
import org.elasticsearch.watcher.trigger.schedule.IntervalSchedule;
|
||||||
|
@ -41,8 +38,6 @@ import org.elasticsearch.watcher.watch.Payload;
|
||||||
import org.elasticsearch.watcher.watch.Watch;
|
import org.elasticsearch.watcher.watch.Watch;
|
||||||
import org.elasticsearch.watcher.watch.WatchStatus;
|
import org.elasticsearch.watcher.watch.WatchStatus;
|
||||||
import org.joda.time.DateTime;
|
import org.joda.time.DateTime;
|
||||||
import org.joda.time.DateTimeZone;
|
|
||||||
import org.joda.time.format.DateTimeFormat;
|
|
||||||
import org.junit.Test;
|
import org.junit.Test;
|
||||||
|
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
|
@ -116,7 +111,7 @@ public class SearchInputTests extends ESIntegTestCase {
|
||||||
.request()
|
.request()
|
||||||
.source(searchSourceBuilder);
|
.source(searchSourceBuilder);
|
||||||
|
|
||||||
ExecutableSearchInput searchInput = new ExecutableSearchInput(new SearchInput(request, null, null, null), logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableSearchInput searchInput = new ExecutableSearchInput(new SearchInput(request, null, null, null), logger, ClientProxy.of(client()), null);
|
||||||
WatchExecutionContext ctx = new TriggeredExecutionContext(
|
WatchExecutionContext ctx = new TriggeredExecutionContext(
|
||||||
new Watch("test-watch",
|
new Watch("test-watch",
|
||||||
new ScheduleTrigger(new IntervalSchedule(new IntervalSchedule.Interval(1, IntervalSchedule.Interval.Unit.MINUTES))),
|
new ScheduleTrigger(new IntervalSchedule(new IntervalSchedule.Interval(1, IntervalSchedule.Interval.Unit.MINUTES))),
|
||||||
|
@ -223,7 +218,7 @@ public class SearchInputTests extends ESIntegTestCase {
|
||||||
.request()
|
.request()
|
||||||
.source(searchSourceBuilder);
|
.source(searchSourceBuilder);
|
||||||
|
|
||||||
ExecutableSearchInput searchInput = new ExecutableSearchInput(new SearchInput(request, null, null, null), logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableSearchInput searchInput = new ExecutableSearchInput(new SearchInput(request, null, null, null), logger, ClientProxy.of(client()), null);
|
||||||
WatchExecutionContext ctx = new TriggeredExecutionContext(
|
WatchExecutionContext ctx = new TriggeredExecutionContext(
|
||||||
new Watch("test-watch",
|
new Watch("test-watch",
|
||||||
new ScheduleTrigger(new IntervalSchedule(new IntervalSchedule.Interval(1, IntervalSchedule.Interval.Unit.MINUTES))),
|
new ScheduleTrigger(new IntervalSchedule(new IntervalSchedule.Interval(1, IntervalSchedule.Interval.Unit.MINUTES))),
|
||||||
|
@ -266,57 +261,6 @@ public class SearchInputTests extends ESIntegTestCase {
|
||||||
assertThat(searchInput.getTimeout(), equalTo(timeout));
|
assertThat(searchInput.getTimeout(), equalTo(timeout));
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testParser_IndexNames() throws Exception {
|
|
||||||
SearchRequest request = client().prepareSearch()
|
|
||||||
.setSearchType(ExecutableSearchInput.DEFAULT_SEARCH_TYPE)
|
|
||||||
.setIndices("test", "<test-{now/d-1d}>")
|
|
||||||
.request()
|
|
||||||
.source(searchSource()
|
|
||||||
.query(boolQuery().must(matchQuery("event_type", "a")).filter(rangeQuery("_timestamp").from("{{ctx.trigger.scheduled_time}}||-30s").to("{{ctx.trigger.triggered_time}}"))));
|
|
||||||
|
|
||||||
DateTime now = DateTime.now(UTC);
|
|
||||||
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.forOffsetHours(-2) : null;
|
|
||||||
if (timeZone != null) {
|
|
||||||
now = now.withHourOfDay(0).withMinuteOfHour(0);
|
|
||||||
}
|
|
||||||
|
|
||||||
boolean timeZoneInWatch = randomBoolean();
|
|
||||||
SearchInput input = timeZone != null && timeZoneInWatch ?
|
|
||||||
new SearchInput(request, null, null, timeZone) :
|
|
||||||
new SearchInput(request, null, null, null);
|
|
||||||
|
|
||||||
XContentBuilder builder = jsonBuilder().value(input);
|
|
||||||
XContentParser parser = JsonXContent.jsonXContent.createParser(builder.bytes());
|
|
||||||
parser.nextToken();
|
|
||||||
|
|
||||||
String dateFormat;
|
|
||||||
Settings.Builder settingsBuilder = Settings.builder();
|
|
||||||
if (randomBoolean()) {
|
|
||||||
dateFormat = DynamicIndexName.DEFAULT_DATE_FORMAT;
|
|
||||||
} else {
|
|
||||||
dateFormat = "YYYY-MM-dd";
|
|
||||||
settingsBuilder.put("watcher.input.search.dynamic_indices.default_date_format", dateFormat);
|
|
||||||
}
|
|
||||||
if (timeZone != null && !timeZoneInWatch) {
|
|
||||||
settingsBuilder.put("watcher.input.search.dynamic_indices.time_zone", timeZone);
|
|
||||||
}
|
|
||||||
|
|
||||||
SearchInputFactory factory = new SearchInputFactory(settingsBuilder.build(), ClientProxy.of(client()));
|
|
||||||
|
|
||||||
ExecutableSearchInput executable = factory.parseExecutable("_id", parser);
|
|
||||||
DynamicIndexName[] indexNames = executable.indexNames();
|
|
||||||
assertThat(indexNames, notNullValue());
|
|
||||||
|
|
||||||
String[] names = DynamicIndexName.names(indexNames, now);
|
|
||||||
assertThat(names, notNullValue());
|
|
||||||
assertThat(names.length, is(2));
|
|
||||||
if (timeZone != null) {
|
|
||||||
now = now.withZone(timeZone);
|
|
||||||
}
|
|
||||||
assertThat(names, arrayContaining("test", "test-" + DateTimeFormat.forPattern(dateFormat).print(now.minusDays(1))));
|
|
||||||
}
|
|
||||||
|
|
||||||
private WatchExecutionContext createContext() {
|
private WatchExecutionContext createContext() {
|
||||||
return new TriggeredExecutionContext(
|
return new TriggeredExecutionContext(
|
||||||
new Watch("test-watch",
|
new Watch("test-watch",
|
||||||
|
@ -340,7 +284,7 @@ public class SearchInputTests extends ESIntegTestCase {
|
||||||
|
|
||||||
SearchInput si = siBuilder.build();
|
SearchInput si = siBuilder.build();
|
||||||
|
|
||||||
ExecutableSearchInput searchInput = new ExecutableSearchInput(si, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableSearchInput searchInput = new ExecutableSearchInput(si, logger, ClientProxy.of(client()), null);
|
||||||
return searchInput.execute(ctx);
|
return searchInput.execute(ctx);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -83,11 +83,12 @@ public class DynamicIndexNameIntegrationTests extends AbstractWatcherIntegration
|
||||||
flush();
|
flush();
|
||||||
refresh();
|
refresh();
|
||||||
|
|
||||||
|
String indexNameDateMathExpressions = "<idx-{now/d}>";
|
||||||
WatcherClient watcherClient = watcherClient();
|
WatcherClient watcherClient = watcherClient();
|
||||||
PutWatchResponse putWatchResponse = watcherClient.preparePutWatch("_id")
|
PutWatchResponse putWatchResponse = watcherClient.preparePutWatch("_id")
|
||||||
.setSource(watchBuilder()
|
.setSource(watchBuilder()
|
||||||
.trigger(schedule(interval(5, IntervalSchedule.Interval.Unit.SECONDS)))
|
.trigger(schedule(interval(5, IntervalSchedule.Interval.Unit.SECONDS)))
|
||||||
.input(searchInput(new SearchRequest("<idx-{now/d}>").types("type"))))
|
.input(searchInput(new SearchRequest(indexNameDateMathExpressions).types("type"))))
|
||||||
.get();
|
.get();
|
||||||
|
|
||||||
assertThat(putWatchResponse.isCreated(), is(true));
|
assertThat(putWatchResponse.isCreated(), is(true));
|
||||||
|
@ -96,23 +97,24 @@ public class DynamicIndexNameIntegrationTests extends AbstractWatcherIntegration
|
||||||
flush();
|
flush();
|
||||||
refresh();
|
refresh();
|
||||||
|
|
||||||
SearchResponse response = searchHistory(searchSource().query(matchQuery("result.input.search.request.indices", indexName)));
|
SearchResponse response = searchHistory(searchSource().query(matchQuery("result.input.search.request.indices", indexNameDateMathExpressions)));
|
||||||
assertThat(response.getHits().getTotalHits(), is(1L));
|
assertThat(response.getHits().getTotalHits(), is(1L));
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
public void testDynamicIndexSearchTransform() throws Exception {
|
public void testDynamicIndexSearchTransform() throws Exception {
|
||||||
final String indexName = "idx-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(timeWarp().clock().nowUTC());
|
String indexName = "idx-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(timeWarp().clock().nowUTC());
|
||||||
createIndex(indexName);
|
createIndex(indexName);
|
||||||
index(indexName, "type", "1", "key", "value");
|
index(indexName, "type", "1", "key", "value");
|
||||||
flush();
|
flush();
|
||||||
refresh();
|
refresh();
|
||||||
|
|
||||||
|
final String indexNameDateMathExpressions = "<idx-{now/d}>";
|
||||||
WatcherClient watcherClient = watcherClient();
|
WatcherClient watcherClient = watcherClient();
|
||||||
PutWatchResponse putWatchResponse = watcherClient.preparePutWatch("_id")
|
PutWatchResponse putWatchResponse = watcherClient.preparePutWatch("_id")
|
||||||
.setSource(watchBuilder()
|
.setSource(watchBuilder()
|
||||||
.trigger(schedule(interval(5, IntervalSchedule.Interval.Unit.SECONDS)))
|
.trigger(schedule(interval(5, IntervalSchedule.Interval.Unit.SECONDS)))
|
||||||
.transform(searchTransform(new SearchRequest("<idx-{now/d}>").types("type")))
|
.transform(searchTransform(new SearchRequest(indexNameDateMathExpressions).types("type")))
|
||||||
.addAction("log", loggingAction("heya")))
|
.addAction("log", loggingAction("heya")))
|
||||||
.get();
|
.get();
|
||||||
|
|
||||||
|
@ -125,7 +127,7 @@ public class DynamicIndexNameIntegrationTests extends AbstractWatcherIntegration
|
||||||
SearchResponse response = searchWatchRecords(new Callback<SearchRequestBuilder>() {
|
SearchResponse response = searchWatchRecords(new Callback<SearchRequestBuilder>() {
|
||||||
@Override
|
@Override
|
||||||
public void handle(SearchRequestBuilder builder) {
|
public void handle(SearchRequestBuilder builder) {
|
||||||
builder.setQuery(matchQuery("result.transform.search.request.indices", indexName));
|
builder.setQuery(matchQuery("result.transform.search.request.indices", indexNameDateMathExpressions));
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
assertThat(response.getHits().getTotalHits(), is(1L));
|
assertThat(response.getHits().getTotalHits(), is(1L));
|
||||||
|
|
|
@ -1,266 +0,0 @@
|
||||||
/*
|
|
||||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
|
||||||
* or more contributor license agreements. Licensed under the Elastic License;
|
|
||||||
* you may not use this file except in compliance with the Elastic License.
|
|
||||||
*/
|
|
||||||
package org.elasticsearch.watcher.support;
|
|
||||||
|
|
||||||
import org.elasticsearch.ElasticsearchParseException;
|
|
||||||
import org.elasticsearch.common.Strings;
|
|
||||||
import org.elasticsearch.common.settings.Settings;
|
|
||||||
import org.elasticsearch.test.ESTestCase;
|
|
||||||
import org.joda.time.DateTime;
|
|
||||||
import org.joda.time.DateTimeZone;
|
|
||||||
import org.joda.time.format.DateTimeFormat;
|
|
||||||
import org.junit.Test;
|
|
||||||
|
|
||||||
import static org.hamcrest.CoreMatchers.nullValue;
|
|
||||||
import static org.hamcrest.Matchers.*;
|
|
||||||
|
|
||||||
/**
|
|
||||||
*
|
|
||||||
*/
|
|
||||||
public class DynamicIndexNameTests extends ESTestCase {
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testNormal() throws Exception {
|
|
||||||
String indexName = randomAsciiOfLength(10);
|
|
||||||
DateTime now = DateTime.now(DateTimeZone.UTC);
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName indexNames = parser.parse(indexName);
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
assertThat(name, equalTo(indexName));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression() throws Exception {
|
|
||||||
DateTime now = DateTime.now(DateTimeZone.UTC);
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName indexNames = parser.parse("<.marvel-{now}>");
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now)));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testNullOrEmpty() throws Exception {
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName indexName = parser.parse((String) null);
|
|
||||||
assertThat(indexName, nullValue());
|
|
||||||
DynamicIndexName[] indexNames = parser.parse(Strings.EMPTY_ARRAY);
|
|
||||||
assertThat(indexNames, nullValue());
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression_Static() throws Exception {
|
|
||||||
DateTime now = DateTime.now(DateTimeZone.UTC);
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName indexNames = parser.parse("<.marvel-test>");
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
assertThat(name, equalTo(".marvel-test"));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression_MultiParts() throws Exception {
|
|
||||||
DateTime now = DateTime.now(DateTimeZone.UTC);
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName indexNames = parser.parse("<.text1-{now/d}-text2-{now/M}>");
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
assertThat(name, equalTo(".text1-"
|
|
||||||
+ DateTimeFormat.forPattern("YYYY.MM.dd").print(now)
|
|
||||||
+ "-text2-"
|
|
||||||
+ DateTimeFormat.forPattern("YYYY.MM.dd").print(now.withDayOfMonth(1))));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression_CustomFormat() throws Exception {
|
|
||||||
DateTime now = DateTime.now(DateTimeZone.UTC);
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName indexNames = parser.parse("<.marvel-{now/d{YYYY.MM.dd}}>");
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now)));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression_CustomTimeZone() throws Exception {
|
|
||||||
DateTimeZone timeZone;
|
|
||||||
int hoursOffset;
|
|
||||||
int minutesOffset = 0;
|
|
||||||
if (randomBoolean()) {
|
|
||||||
hoursOffset = randomIntBetween(-12, 14);
|
|
||||||
timeZone = DateTimeZone.forOffsetHours(hoursOffset);
|
|
||||||
} else {
|
|
||||||
hoursOffset = randomIntBetween(-11, 13);
|
|
||||||
minutesOffset = randomIntBetween(0, 59);
|
|
||||||
timeZone = DateTimeZone.forOffsetHoursMinutes(hoursOffset, minutesOffset);
|
|
||||||
}
|
|
||||||
DateTime now;
|
|
||||||
if (hoursOffset >= 0) {
|
|
||||||
// rounding to next day 00:00
|
|
||||||
now = DateTime.now(DateTimeZone.UTC).plusHours(hoursOffset).plusMinutes(minutesOffset).withHourOfDay(0).withMinuteOfHour(0).withSecondOfMinute(0);
|
|
||||||
} else {
|
|
||||||
// rounding to today 00:00
|
|
||||||
now = DateTime.now(DateTimeZone.UTC).withHourOfDay(0).withMinuteOfHour(0).withSecondOfMinute(0);
|
|
||||||
}
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", timeZone);
|
|
||||||
DynamicIndexName indexNames = parser.parse("<.marvel-{now/d{YYYY.MM.dd}}>");
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
logger.info("timezone: [{}], now [{}], name: [{}]", timeZone, now, name);
|
|
||||||
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.withZone(timeZone))));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression_CustomTimeZone_OnParse() throws Exception {
|
|
||||||
DateTimeZone timeZone;
|
|
||||||
int hoursOffset;
|
|
||||||
int minutesOffset = 0;
|
|
||||||
if (randomBoolean()) {
|
|
||||||
hoursOffset = randomIntBetween(-12, 14);
|
|
||||||
timeZone = DateTimeZone.forOffsetHours(hoursOffset);
|
|
||||||
} else {
|
|
||||||
hoursOffset = randomIntBetween(-11, 13);
|
|
||||||
minutesOffset = randomIntBetween(0, 59);
|
|
||||||
timeZone = DateTimeZone.forOffsetHoursMinutes(hoursOffset, minutesOffset);
|
|
||||||
}
|
|
||||||
DateTime now;
|
|
||||||
if (hoursOffset >= 0) {
|
|
||||||
// rounding to next day 00:00
|
|
||||||
now = DateTime.now(DateTimeZone.UTC).plusHours(hoursOffset).plusMinutes(minutesOffset).withHourOfDay(0).withMinuteOfHour(0).withSecondOfMinute(0);
|
|
||||||
} else {
|
|
||||||
// rounding to today 00:00
|
|
||||||
now = DateTime.now(DateTimeZone.UTC).withHourOfDay(0).withMinuteOfHour(0).withSecondOfMinute(0);
|
|
||||||
}
|
|
||||||
Settings settings = Settings.builder()
|
|
||||||
.put("watcher.dynamic_indices.default_date_format", "YYYY.MM.dd")
|
|
||||||
.put("watcher.dynamic_indices.time_zone", "-12")
|
|
||||||
.put("watcher.foo.dynamic_indices.time_zone", "-12")
|
|
||||||
.build();
|
|
||||||
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser(settings, "watcher.foo");
|
|
||||||
DynamicIndexName indexNames = parser.parse("<.marvel-{now/d{YYYY.MM.dd}}>", timeZone);
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
logger.info("timezone: [{}], now [{}], name: [{}]", timeZone, now, name);
|
|
||||||
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.withZone(timeZone))));
|
|
||||||
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression_EscapeStatic() throws Exception {
|
|
||||||
DateTime now = DateTime.now(DateTimeZone.UTC);
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName indexNames = parser.parse("<.mar\\{v\\}el-{now/d}>");
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
assertThat(name, equalTo(".mar{v}el-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now)));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression_EscapeDateFormat() throws Exception {
|
|
||||||
DateTime now = DateTime.now(DateTimeZone.UTC);
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName indexNames = parser.parse("<.marvel-{now/d{'\\{year\\}'YYYY}}>");
|
|
||||||
String name = indexNames.name(now);
|
|
||||||
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("'{year}'YYYY").print(now)));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testExpression_MixedArray() throws Exception {
|
|
||||||
DateTime now = DateTime.now(DateTimeZone.UTC);
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
DynamicIndexName[] indexNames = parser.parse(new String[] {
|
|
||||||
"name1",
|
|
||||||
"<.marvel-{now/d}>",
|
|
||||||
"name2",
|
|
||||||
"<.logstash-{now/M{YYYY.MM}}>"
|
|
||||||
});
|
|
||||||
String[] names = new String[indexNames.length];
|
|
||||||
for (int i = 0; i < names.length; i++) {
|
|
||||||
names[i] = indexNames[i].name(now);
|
|
||||||
}
|
|
||||||
assertThat(names.length, is(4));
|
|
||||||
assertThat(names, arrayContaining(
|
|
||||||
"name1",
|
|
||||||
".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now),
|
|
||||||
"name2",
|
|
||||||
".logstash-" + DateTimeFormat.forPattern("YYYY.MM").print(now.withDayOfMonth(1))));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test(expected = ElasticsearchParseException.class)
|
|
||||||
public void testExpression_Invalid_Unescaped() throws Exception {
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
parser.parse("<.mar}vel-{now/d}>");
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test(expected = ElasticsearchParseException.class)
|
|
||||||
public void testExpression_Invalid_DateMathFormat() throws Exception {
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
parser.parse("<.marvel-{now/d{}>");
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test(expected = ElasticsearchParseException.class)
|
|
||||||
public void testExpression_Invalid_EmptyDateMathFormat() throws Exception {
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
parser.parse("<.marvel-{now/d{}}>");
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test(expected = ElasticsearchParseException.class)
|
|
||||||
public void testExpression_Invalid_OpenEnded() throws Exception {
|
|
||||||
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
|
|
||||||
parser.parse("<.marvel-{now/d>");
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testDefaultDateFormat_Default() throws Exception {
|
|
||||||
String dateFormat = DynamicIndexName.defaultDateFormat(Settings.EMPTY);
|
|
||||||
assertThat(dateFormat, is("YYYY.MM.dd"));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testDefaultDateFormat() throws Exception {
|
|
||||||
Settings settings = Settings.builder()
|
|
||||||
.put("watcher.dynamic_indices.default_date_format", "YYYY.MM")
|
|
||||||
.build();
|
|
||||||
String dateFormat = randomBoolean() ?
|
|
||||||
DynamicIndexName.defaultDateFormat(settings) :
|
|
||||||
DynamicIndexName.defaultDateFormat(settings, null);
|
|
||||||
assertThat(dateFormat, is("YYYY.MM"));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testDefaultDateFormat_Component() throws Exception {
|
|
||||||
Settings settings = Settings.builder()
|
|
||||||
.put("watcher.dynamic_indices.default_date_format", "YYYY.MM")
|
|
||||||
.put("watcher.foo.dynamic_indices.default_date_format", "YYY.MM")
|
|
||||||
.build();
|
|
||||||
String dateFormat = DynamicIndexName.defaultDateFormat(settings, "watcher.foo");
|
|
||||||
assertThat(dateFormat, is("YYY.MM"));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testTimeZone_Default() throws Exception {
|
|
||||||
DateTimeZone timeZone = DynamicIndexName.timeZone(Settings.EMPTY);
|
|
||||||
assertThat(timeZone, is(DateTimeZone.UTC));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testTimeZone() throws Exception {
|
|
||||||
DateTimeZone timeZone = DateTimeZone.forOffsetHours(randomIntBetween(-12, 14));
|
|
||||||
Settings settings = Settings.builder()
|
|
||||||
.put("watcher.dynamic_indices.time_zone", timeZone)
|
|
||||||
.build();
|
|
||||||
DateTimeZone resolvedTimeZone = randomBoolean() ?
|
|
||||||
DynamicIndexName.timeZone(settings) :
|
|
||||||
DynamicIndexName.timeZone(settings, null);
|
|
||||||
assertThat(timeZone, is(resolvedTimeZone));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testTimeZone_Component() throws Exception {
|
|
||||||
DateTimeZone timeZone = DateTimeZone.forOffsetHours(randomIntBetween(-11, 14));
|
|
||||||
Settings settings = Settings.builder()
|
|
||||||
.put("watcher.dynamic_indices.time_zone", "-12")
|
|
||||||
.put("watcher.foo.dynamic_indices.time_zone", timeZone)
|
|
||||||
.build();
|
|
||||||
DateTimeZone resolvedTimeZone = DynamicIndexName.timeZone(settings, "watcher.foo");
|
|
||||||
assertThat(resolvedTimeZone, is(timeZone));
|
|
||||||
}
|
|
||||||
|
|
||||||
}
|
|
|
@ -41,7 +41,6 @@ import org.elasticsearch.watcher.input.search.ExecutableSearchInput;
|
||||||
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
||||||
import org.elasticsearch.watcher.input.simple.SimpleInput;
|
import org.elasticsearch.watcher.input.simple.SimpleInput;
|
||||||
import org.elasticsearch.watcher.license.LicenseService;
|
import org.elasticsearch.watcher.license.LicenseService;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.Script;
|
import org.elasticsearch.watcher.support.Script;
|
||||||
import org.elasticsearch.watcher.support.WatcherUtils;
|
import org.elasticsearch.watcher.support.WatcherUtils;
|
||||||
import org.elasticsearch.watcher.support.http.HttpClient;
|
import org.elasticsearch.watcher.support.http.HttpClient;
|
||||||
|
@ -69,7 +68,6 @@ import org.joda.time.DateTime;
|
||||||
|
|
||||||
import javax.mail.internet.AddressException;
|
import javax.mail.internet.AddressException;
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
import java.lang.reflect.Constructor;
|
|
||||||
import java.nio.charset.StandardCharsets;
|
import java.nio.charset.StandardCharsets;
|
||||||
import java.util.*;
|
import java.util.*;
|
||||||
|
|
||||||
|
@ -222,7 +220,7 @@ public final class WatcherTestUtils {
|
||||||
new ScheduleTrigger(new CronSchedule("0/5 * * * * ? *")),
|
new ScheduleTrigger(new CronSchedule("0/5 * * * * ? *")),
|
||||||
new ExecutableSimpleInput(new SimpleInput(new Payload.Simple(inputData)), logger),
|
new ExecutableSimpleInput(new SimpleInput(new Payload.Simple(inputData)), logger),
|
||||||
new ExecutableScriptCondition(new ScriptCondition(Script.inline("return true").build()), logger, scriptService),
|
new ExecutableScriptCondition(new ScriptCondition(Script.inline("return true").build()), logger, scriptService),
|
||||||
new ExecutableSearchTransform(new SearchTransform(transformRequest, null, null), logger, client, null, new DynamicIndexName.Parser()),
|
new ExecutableSearchTransform(new SearchTransform(transformRequest, null, null), logger, client, null),
|
||||||
new TimeValue(0),
|
new TimeValue(0),
|
||||||
new ExecutableActions(actions),
|
new ExecutableActions(actions),
|
||||||
metadata,
|
metadata,
|
||||||
|
|
|
@ -30,7 +30,6 @@ import org.elasticsearch.watcher.execution.TriggeredExecutionContext;
|
||||||
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
import org.elasticsearch.watcher.execution.WatchExecutionContext;
|
||||||
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
||||||
import org.elasticsearch.watcher.input.simple.SimpleInput;
|
import org.elasticsearch.watcher.input.simple.SimpleInput;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
|
||||||
import org.elasticsearch.watcher.support.text.TextTemplate;
|
import org.elasticsearch.watcher.support.text.TextTemplate;
|
||||||
import org.elasticsearch.watcher.transform.Transform;
|
import org.elasticsearch.watcher.transform.Transform;
|
||||||
|
@ -42,8 +41,6 @@ import org.elasticsearch.watcher.watch.Payload;
|
||||||
import org.elasticsearch.watcher.watch.Watch;
|
import org.elasticsearch.watcher.watch.Watch;
|
||||||
import org.elasticsearch.watcher.watch.WatchStatus;
|
import org.elasticsearch.watcher.watch.WatchStatus;
|
||||||
import org.joda.time.DateTime;
|
import org.joda.time.DateTime;
|
||||||
import org.joda.time.DateTimeZone;
|
|
||||||
import org.joda.time.format.DateTimeFormat;
|
|
||||||
import org.junit.Test;
|
import org.junit.Test;
|
||||||
|
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
|
@ -128,7 +125,7 @@ public class SearchTransformTests extends ESIntegTestCase {
|
||||||
.endObject()
|
.endObject()
|
||||||
.endObject().bytes());
|
.endObject().bytes());
|
||||||
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
|
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
|
||||||
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null);
|
||||||
|
|
||||||
WatchExecutionContext ctx = mockExecutionContext("_name", EMPTY_PAYLOAD);
|
WatchExecutionContext ctx = mockExecutionContext("_name", EMPTY_PAYLOAD);
|
||||||
|
|
||||||
|
@ -165,7 +162,7 @@ public class SearchTransformTests extends ESIntegTestCase {
|
||||||
.endObject()
|
.endObject()
|
||||||
.endObject().bytes());
|
.endObject().bytes());
|
||||||
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
|
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
|
||||||
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null);
|
||||||
|
|
||||||
WatchExecutionContext ctx = mockExecutionContext("_name", EMPTY_PAYLOAD);
|
WatchExecutionContext ctx = mockExecutionContext("_name", EMPTY_PAYLOAD);
|
||||||
|
|
||||||
|
@ -211,7 +208,7 @@ public class SearchTransformTests extends ESIntegTestCase {
|
||||||
.must(termQuery("value", "{{ctx.payload.value}}"))));
|
.must(termQuery("value", "{{ctx.payload.value}}"))));
|
||||||
|
|
||||||
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
|
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
|
||||||
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null);
|
||||||
|
|
||||||
ScheduleTriggerEvent event = new ScheduleTriggerEvent("_name", parseDate("2015-01-04T00:00:00", UTC), parseDate("2015-01-01T00:00:00", UTC));
|
ScheduleTriggerEvent event = new ScheduleTriggerEvent("_name", parseDate("2015-01-04T00:00:00", UTC), parseDate("2015-01-01T00:00:00", UTC));
|
||||||
WatchExecutionContext ctx = mockExecutionContext("_name", parseDate("2015-01-04T00:00:00", UTC), event, EMPTY_PAYLOAD);
|
WatchExecutionContext ctx = mockExecutionContext("_name", parseDate("2015-01-04T00:00:00", UTC), event, EMPTY_PAYLOAD);
|
||||||
|
@ -303,68 +300,6 @@ public class SearchTransformTests extends ESIntegTestCase {
|
||||||
assertThat(executable.transform().getTimeout(), equalTo(readTimeout));
|
assertThat(executable.transform().getTimeout(), equalTo(readTimeout));
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testParser_WithIndexNames() throws Exception {
|
|
||||||
SearchType searchType = getRandomSupportedSearchType();
|
|
||||||
XContentBuilder builder = jsonBuilder().startObject();
|
|
||||||
builder.startObject("request");
|
|
||||||
builder.array("indices", "idx", "<idx-{now/d-3d}>");
|
|
||||||
if (searchType != null) {
|
|
||||||
builder.field("search_type", searchType.name());
|
|
||||||
}
|
|
||||||
|
|
||||||
DateTime now = DateTime.now(UTC);
|
|
||||||
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.forOffsetHours(-2) : null;
|
|
||||||
if (timeZone != null) {
|
|
||||||
now = now.withHourOfDay(0).withMinuteOfHour(0);
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.startObject("body")
|
|
||||||
.startObject("query")
|
|
||||||
.startObject("match_all")
|
|
||||||
.endObject()
|
|
||||||
.endObject()
|
|
||||||
.endObject();
|
|
||||||
builder.endObject();
|
|
||||||
|
|
||||||
boolean timeZoneInWatch = randomBoolean();
|
|
||||||
if (timeZone != null && timeZoneInWatch) {
|
|
||||||
builder.field(SearchTransform.Field.DYNAMIC_NAME_TIMEZONE.getPreferredName(), timeZone);
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.endObject();
|
|
||||||
|
|
||||||
XContentParser parser = JsonXContent.jsonXContent.createParser(builder.bytes());
|
|
||||||
parser.nextToken();
|
|
||||||
|
|
||||||
String dateFormat;
|
|
||||||
Settings.Builder settingsBuilder = Settings.builder();
|
|
||||||
if (randomBoolean()) {
|
|
||||||
dateFormat = DynamicIndexName.DEFAULT_DATE_FORMAT;
|
|
||||||
} else {
|
|
||||||
dateFormat = "YYYY-MM";
|
|
||||||
settingsBuilder.put("watcher.transform.search.dynamic_indices.default_date_format", dateFormat);
|
|
||||||
}
|
|
||||||
if (timeZone != null && !timeZoneInWatch) {
|
|
||||||
settingsBuilder.put("watcher.transform.search.dynamic_indices.time_zone", timeZone);
|
|
||||||
}
|
|
||||||
Settings settings = settingsBuilder.build();
|
|
||||||
|
|
||||||
SearchTransformFactory transformFactory = new SearchTransformFactory(settings, ClientProxy.of(client()));
|
|
||||||
|
|
||||||
ExecutableSearchTransform executable = transformFactory.parseExecutable("_id", parser);
|
|
||||||
DynamicIndexName[] indexNames = executable.indexNames();
|
|
||||||
assertThat(indexNames, notNullValue());
|
|
||||||
|
|
||||||
String[] names = DynamicIndexName.names(indexNames, now);
|
|
||||||
assertThat(names, notNullValue());
|
|
||||||
assertThat(names.length, is(2));
|
|
||||||
if (timeZone != null) {
|
|
||||||
now = now.withZone(timeZone);
|
|
||||||
}
|
|
||||||
assertThat(names, arrayContaining("idx", "idx-" + DateTimeFormat.forPattern(dateFormat).print(now.minusDays(3))));
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
public void testSearch_InlineTemplate() throws Exception {
|
public void testSearch_InlineTemplate() throws Exception {
|
||||||
WatchExecutionContext ctx = createContext();
|
WatchExecutionContext ctx = createContext();
|
||||||
|
@ -501,7 +436,7 @@ public class SearchTransformTests extends ESIntegTestCase {
|
||||||
ensureGreen("test-search-index");
|
ensureGreen("test-search-index");
|
||||||
|
|
||||||
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
|
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
|
||||||
ExecutableSearchTransform executableSearchTransform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
|
ExecutableSearchTransform executableSearchTransform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null);
|
||||||
|
|
||||||
return executableSearchTransform.execute(ctx, Payload.Simple.EMPTY);
|
return executableSearchTransform.execute(ctx, Payload.Simple.EMPTY);
|
||||||
}
|
}
|
||||||
|
|
|
@ -60,7 +60,6 @@ import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
|
||||||
import org.elasticsearch.watcher.input.simple.SimpleInput;
|
import org.elasticsearch.watcher.input.simple.SimpleInput;
|
||||||
import org.elasticsearch.watcher.input.simple.SimpleInputFactory;
|
import org.elasticsearch.watcher.input.simple.SimpleInputFactory;
|
||||||
import org.elasticsearch.watcher.license.LicenseService;
|
import org.elasticsearch.watcher.license.LicenseService;
|
||||||
import org.elasticsearch.watcher.support.DynamicIndexName;
|
|
||||||
import org.elasticsearch.watcher.support.Script;
|
import org.elasticsearch.watcher.support.Script;
|
||||||
import org.elasticsearch.watcher.support.WatcherUtils;
|
import org.elasticsearch.watcher.support.WatcherUtils;
|
||||||
import org.elasticsearch.watcher.support.clock.Clock;
|
import org.elasticsearch.watcher.support.clock.Clock;
|
||||||
|
@ -121,7 +120,6 @@ public class WatchTests extends ESTestCase {
|
||||||
private SecretService secretService;
|
private SecretService secretService;
|
||||||
private LicenseService licenseService;
|
private LicenseService licenseService;
|
||||||
private ESLogger logger;
|
private ESLogger logger;
|
||||||
private DynamicIndexName.Parser indexNameParser;
|
|
||||||
private Settings settings = Settings.EMPTY;
|
private Settings settings = Settings.EMPTY;
|
||||||
|
|
||||||
@Before
|
@Before
|
||||||
|
@ -136,7 +134,6 @@ public class WatchTests extends ESTestCase {
|
||||||
licenseService = mock(LicenseService.class);
|
licenseService = mock(LicenseService.class);
|
||||||
authRegistry = new HttpAuthRegistry(ImmutableMap.of("basic", (HttpAuthFactory) new BasicAuthFactory(secretService)));
|
authRegistry = new HttpAuthRegistry(ImmutableMap.of("basic", (HttpAuthFactory) new BasicAuthFactory(secretService)));
|
||||||
logger = Loggers.getLogger(WatchTests.class);
|
logger = Loggers.getLogger(WatchTests.class);
|
||||||
indexNameParser = new DynamicIndexName.Parser();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
|
@ -311,7 +308,7 @@ public class WatchTests extends ESTestCase {
|
||||||
switch (type) {
|
switch (type) {
|
||||||
case SearchInput.TYPE:
|
case SearchInput.TYPE:
|
||||||
SearchInput searchInput = searchInput(WatcherTestUtils.newInputSearchRequest("idx")).build();
|
SearchInput searchInput = searchInput(WatcherTestUtils.newInputSearchRequest("idx")).build();
|
||||||
return new ExecutableSearchInput(searchInput, logger, client, null, indexNameParser);
|
return new ExecutableSearchInput(searchInput, logger, client, null);
|
||||||
default:
|
default:
|
||||||
SimpleInput simpleInput = InputBuilders.simpleInput(ImmutableMap.<String, Object>builder().put("_key", "_val")).build();
|
SimpleInput simpleInput = InputBuilders.simpleInput(ImmutableMap.<String, Object>builder().put("_key", "_val")).build();
|
||||||
return new ExecutableSimpleInput(simpleInput, logger);
|
return new ExecutableSimpleInput(simpleInput, logger);
|
||||||
|
@ -370,12 +367,12 @@ public class WatchTests extends ESTestCase {
|
||||||
case ScriptTransform.TYPE:
|
case ScriptTransform.TYPE:
|
||||||
return new ExecutableScriptTransform(new ScriptTransform(Script.inline("_script").build()), logger, scriptService);
|
return new ExecutableScriptTransform(new ScriptTransform(Script.inline("_script").build()), logger, scriptService);
|
||||||
case SearchTransform.TYPE:
|
case SearchTransform.TYPE:
|
||||||
return new ExecutableSearchTransform(new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), logger, client, null, indexNameParser);
|
return new ExecutableSearchTransform(new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), logger, client, null);
|
||||||
default: // chain
|
default: // chain
|
||||||
ChainTransform chainTransform = new ChainTransform(Arrays.asList(
|
ChainTransform chainTransform = new ChainTransform(Arrays.asList(
|
||||||
new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), new ScriptTransform(Script.inline("_script").build())));
|
new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), new ScriptTransform(Script.inline("_script").build())));
|
||||||
return new ExecutableChainTransform(chainTransform, logger, Arrays.<ExecutableTransform>asList(
|
return new ExecutableChainTransform(chainTransform, logger, Arrays.<ExecutableTransform>asList(
|
||||||
new ExecutableSearchTransform(new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), logger, client, null, indexNameParser),
|
new ExecutableSearchTransform(new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), logger, client, null),
|
||||||
new ExecutableScriptTransform(new ScriptTransform(Script.inline("_script").build()), logger, scriptService)));
|
new ExecutableScriptTransform(new ScriptTransform(Script.inline("_script").build()), logger, scriptService)));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -402,7 +399,7 @@ public class WatchTests extends ESTestCase {
|
||||||
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.UTC : null;
|
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.UTC : null;
|
||||||
TimeValue timeout = randomBoolean() ? TimeValue.timeValueSeconds(30) : null;
|
TimeValue timeout = randomBoolean() ? TimeValue.timeValueSeconds(30) : null;
|
||||||
IndexAction action = new IndexAction("_index", "_type", null, timeout, timeZone);
|
IndexAction action = new IndexAction("_index", "_type", null, timeout, timeZone);
|
||||||
list.add(new ActionWrapper("_index_" + randomAsciiOfLength(8), randomThrottler(), randomTransform(), new ExecutableIndexAction(action, logger, client, null, indexNameParser)));
|
list.add(new ActionWrapper("_index_" + randomAsciiOfLength(8), randomThrottler(), randomTransform(), new ExecutableIndexAction(action, logger, client, null)));
|
||||||
}
|
}
|
||||||
if (randomBoolean()) {
|
if (randomBoolean()) {
|
||||||
HttpRequestTemplate httpRequest = HttpRequestTemplate.builder("test.host", randomIntBetween(8000, 9000))
|
HttpRequestTemplate httpRequest = HttpRequestTemplate.builder("test.host", randomIntBetween(8000, 9000))
|
||||||
|
|
Loading…
Reference in New Issue