honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Removed dynamic index names in favour for ES' date math index names. 

Original commit: elastic/x-pack-elasticsearch@267084f163
This commit is contained in:
Martijn van Groningen 2015-09-15 11:10:23 +02:00
parent 7fb98b60b4
commit e16894fb9d
21 changed files with 52 additions and 961 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
watcher/docs/how-watcher-works.asciidoc
View File

@ -423,5 +423,3 @@ name in the `file` field. For example, the following snippet references the scri
}
}
--------------------------------------------------
include::how-watcher-works/dynamic-index-names.asciidoc[]

106
watcher/docs/how-watcher-works/dynamic-index-names.asciidoc
View File

@ -1,106 +0,0 @@
[[dynamic-index-names]]
=== Dynamic Index Names
Several watch constructs deal with indices, including <<actions-index, `index` action>>,
the <<transform-search, `search` transform>> and the <<input-search, `search` input>>.
When configuring these constructs you can set the index names to static values. In addition
to specifying static index names, Watcher enables you to specify indexes using dynamic
time-aware templates. These templates resolve to specific index names during the watch
execution according to the execution time.
Dynamic index name resolution enables you to search a range of time-series indices, rather
than searching all of your time-series indices and filtering the the results. Limiting the
number of indices that are searched reduces the load on the cluster and improves watch
execution performance. For example, if you are using a watch to monitor errors in your
daily logs, you can use a dynamic index name template to restrict the search to the past
two days.
A dynamic index name takes the following form:
[source,txt]
----------------------------------------------------------------------
<static_name{date_math_expr{date_format}}>
----------------------------------------------------------------------
Where:
* `static_name` is the static text part of the name
* `date_math_expr` is a dynamic date math expression that computes the date dynamically
* `date_format` is the format in which the computed date should be rendered
NOTE: You must enclose dynamic index name templates within angle brackets. For example,
`<logstash-{now/d-2d}>`
The following example shows different forms of dynamic index names and the final index names
they resolve to given the execution date is 22rd March 2024.
[options="header"]
|======
| Expression |Resolves to
| `<logstash-{now/d}>` | `logstash-2024.03.22`
| `<logstash-{now/M}>` | `logstash-2024.03.01`
| `<logstash-{now/M{YYYY.MM}}>` | `logstash-2024.03`
| `<logstash-{now/M-1M{YYYY.MM}}>` | `logstash-2024.02`
|======
To use the characters `{` and `}` in the static part of an index name template, escape them
with a backslash, `\`:
* `<elastic\\{ON\\}-{now/M}>` resolves to `elastic{ON}-2024.03.01`
The following example shows a search input that searches the Logstash indices for the past
three days, assuming the indices use the default Logstash index name format,
`logstash-YYYY.MM.dd`.
[source,json]
----------------------------------------------------------------------
{
...
"input" : {
"search" : {
"request" : {
"indices" : [
"<logstash-{now/d-2d}>",
"<logstash-{now/d-1d}>",
"<logstash-{now/d}>"
],
...
}
}
}
...
}
----------------------------------------------------------------------
[[dynamic-index-name-timezone]]
By default, the index names are resolved base on `UTC` time zone. You can change this default at
multiple levels:
Configuring the following setting set the default dynamic index name time zone in watcher:
[source,yaml]
--------------------------------------------------
watcher.dynamic_indices.time_zone: '+01:00'
--------------------------------------------------
You can also configure the default time zone separately on each of the construct that make
use of it (`search` input/transform and `index` action):
[source,yaml]
--------------------------------------------------
watcher.input.search.dynamic_indices.time_zone: '+01:00'
--------------------------------------------------
[source,yaml]
--------------------------------------------------
watcher.transform.search.dynamic_indices.time_zone: '+01:00'
--------------------------------------------------
[source,yaml]
--------------------------------------------------
watcher.actions.index.dynamic_indices.time_zone: '+01:00'
--------------------------------------------------
Alternatively, each of these construct can define their own time zone within the watch
definition.

6
watcher/docs/reference/actions/index.asciidoc
View File

@ -33,7 +33,7 @@ The following snippet shows a simple `index` action definition:
|Name |Required | Default | Description
| `index` | yes | - | The Elasticsearch index to
index into. <<dynamic-index-names, Dynamic index names>>
index into.
are supported
| `doc_type` | yes | - | The type of the document
@ -51,10 +51,6 @@ The following snippet shows a simple `index` action definition:
the default internal index/bulk operations
<<default-internal-ops-timeouts, timeouts>>.
| `dynamic_name_timezone` | no | - | The time zone to use for resolving the index name based on
<<dynamic-index-names, Dynamic Index Names>>. The default
time zone also can be <<dynamic-index-name-timezone, configured>>
globally.
|======

3
watcher/docs/reference/input/search.asciidoc
View File

@ -19,7 +19,7 @@ example, to get the message field from the first hit, use `ctx.payload.hits.hits
|======
| Name |Required | Default | Description
| `request.search_type` | no | count | The {ref}/search-request-search-type.html#search-request-search-type[type] of search request to perform. Valid values are: `count`, `dfs_query_and_fetch`, `dfs_query_then_fetch`, `query_and_fetch`, `query_then_fetch`, and `scan`. The Elasticsearch default is `query_then_fetch`.
| `request.indices` | no | - | The indices to search. If omitted, all indices are searched, which is the default behaviour in Elasticsearch. <<dynamic-index-names, Dynamic index names>> are supported.
| `request.indices` | no | - | The indices to search. If omitted, all indices are searched, which is the default behaviour in Elasticsearch.
| `request.types` | no | - | The document types to search for. If omitted, all document types are are searched, which is the default behaviour in Elasticsearch.
| `request.body` | no | - | The body of the request. The {ref}/search-request-body.html[request body] follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` <<templates, templates>>.
| `request.template` | no | - | The body of the search template. See <<templates, configure templates>> for more information.
@ -29,7 +29,6 @@ example, to get the message field from the first hit, use `ctx.payload.hits.hits
| `extract` | no | - | A array of JSON keys to extract from the search response and load as the payload. When a search generates a large response, you can use `extract` to select the relevant fields instead of loading the entire response.
| `timeout` | no | 30s | The timeout for waiting for the search api call to return. If no response is returned within this time, the search input times out and fails.
This setting overrides the default internal search operations <<default-internal-ops-timeouts, timeouts>>.
| `dynamic_name_timezone` | no | - | The time zone to use for resolving the index name based on <<dynamic-index-names, Dynamic Index Names>>. The default time zone also can be <<dynamic-index-name-timezone, configured>> globally.
|======
You can reference the following variables in the execution context when specifying the request `body`:

3
watcher/docs/reference/transform/search.asciidoc
View File

@ -56,7 +56,7 @@ The following table lists all available settings for the search transform:
|======
| Name |Required | Default | Description
| `request.search_type` | no | {ref}/search-request-search-type.html#query-then-fetch[query_then_fetch] | The search {ref}/search-request-search-type.html[search type]
| `request.indices` | no | all indices | One or more indices to search on (may be a comma-delimited string or an array of indices names). <<dynamic-index-names, Dynamic index names>> are supported.
| `request.indices` | no | all indices | One or more indices to search on (may be a comma-delimited string or an array of indices names).
| `request.types` | no | all types | One or more document types to search on (may be a comma-delimited string or an array of document types names)
| `request.body` | no | `match_all` query | The body of the request. The {ref}/search-request-body.html[request body] follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` <<templates, templates>>.
| `request.indices_options.expand_wildcards` | no | `open` | Determines how to expand indices wildcards. Can be one of `open`, `closed`, `none` or `all` (see {ref}/multi-index.html[multi-index support])
@ -65,7 +65,6 @@ The following table lists all available settings for the search transform:
| `request.template` | no | - | The body of the search template. See <<templates, configure templates>> for more information.
| `timeout` | no | 30s | The timeout for waiting for the search api call to return. If no response is returned within this time, the search transform times out and fails.
This setting overrides the default internal search operations <<default-internal-ops-timeouts, timeouts>>.
| `dynamic_name_timezone` | no | - | The time zone to use for resolving the index name based on <<dynamic-index-names, Dynamic Index Names>>. The default time zone also can be <<dynamic-index-name-timezone, configured>> globally.
|======
[[transform-search-template]]

17
watcher/docs/release-notes.asciidoc
View File

@ -36,6 +36,21 @@ bin/plugin remove watcher
[[change-list]]
=== Change List
[float]
==== 2.0.0-rc1
.Breaking
* The dynamic index names support has been removed and Elasticsearch's date math index names support should be used instead.
The only difference between Watcher's dynamic index names support and Elasticsearch's date math index names support is
how timezones are expressed. In Watcher this is done via node settings, in Elasticsearch the timezone is part of the
date math index names support. Only if you're using dynamic index names with timezones in Watcher then you need to
upgrade your watches after the upgrade, otherwise your watches will work as they did before the upgrade. For example if
`watcher.dynamic_indices.time_zone` setting was set to `+01:00` and a watch has the following index name `<logstash-{now/d}>`
then after the upgrade you need to update this watch to use the following index name `<logstash-{now/d{YYYY.MM.dd|+01:00}}>`.
.Bug Fixes
* Fixed url encoding issue in http input and webhook output. The url params were url encoded twice.
[float]
==== 2.0.0-beta2
@ -73,7 +88,7 @@ been aligned with the Elasticsearch versioning.
==== 1.0.0
.Enhancements
* Added execution time aware <<dynamic-index-names, dynamic index names>> support to `index`
* Added execution time aware dynamic index names support to `index`
action, `search` input, and `search` transform.
* You must now explicitly specify the unit when configuring any time value. (Numeric-only
values are no longer supported.)

13
watcher/src/main/java/org/elasticsearch/watcher/actions/index/ExecutableIndexAction.java
View File

@ -20,7 +20,6 @@ import org.elasticsearch.watcher.actions.Action;
import org.elasticsearch.watcher.actions.ExecutableAction;
import org.elasticsearch.watcher.execution.WatchExecutionContext;
import org.elasticsearch.watcher.support.ArrayObjectIterator;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.WatcherDateTimeUtils;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
import org.elasticsearch.watcher.support.xcontent.XContentSource;
@ -37,17 +36,11 @@ public class ExecutableIndexAction extends ExecutableAction<IndexAction> {
private final ClientProxy client;
private final TimeValue timeout;
private final DynamicIndexName indexName;
public ExecutableIndexAction(IndexAction action, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout, DynamicIndexName.Parser indexNameParser) {
public ExecutableIndexAction(IndexAction action, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout) {
super(action, logger);
this.client = client;
this.timeout = action.timeout != null ? action.timeout : defaultTimeout;
this.indexName = indexNameParser.parse(action.index, action.dynamicNameTimeZone);
}
DynamicIndexName indexName() {
return indexName;
}
@Override
@ -71,7 +64,7 @@ public class ExecutableIndexAction extends ExecutableAction<IndexAction> {
IndexRequest indexRequest = new IndexRequest();
indexRequest.index(indexName.name(ctx.executionTime()));
indexRequest.index(action.index);
indexRequest.type(action.docType);
if (action.executionTimeField != null && !TimestampFieldMapper.NAME.equals(action.executionTimeField)) {
@ -104,7 +97,7 @@ public class ExecutableIndexAction extends ExecutableAction<IndexAction> {
}
Map<String, Object> doc = (Map<String, Object>) item;
IndexRequest indexRequest = new IndexRequest();
indexRequest.index(indexName.name(ctx.executionTime()));
indexRequest.index(action.index);
indexRequest.type(action.docType);
if (action.executionTimeField != null && !TimestampFieldMapper.NAME.equals(action.executionTimeField)) {
if (!(doc instanceof HashMap)) {

5
watcher/src/main/java/org/elasticsearch/watcher/actions/index/IndexActionFactory.java
View File

@ -12,7 +12,6 @@ import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.watcher.actions.ActionFactory;
import org.elasticsearch.watcher.actions.email.ExecutableEmailAction;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
import java.io.IOException;
@ -23,14 +22,12 @@ import java.io.IOException;
public class IndexActionFactory extends ActionFactory<IndexAction, ExecutableIndexAction> {
private final ClientProxy client;
private final DynamicIndexName.Parser indexNamesParser;
private final TimeValue defaultTimeout;
@Inject
public IndexActionFactory(Settings settings, ClientProxy client) {
super(Loggers.getLogger(ExecutableEmailAction.class, settings));
this.client = client;
this.indexNamesParser = new DynamicIndexName.Parser(settings, "watcher.actions.index");
this.defaultTimeout = settings.getAsTime("watcher.actions.index.default_timeout", null);
}
@ -46,6 +43,6 @@ public class IndexActionFactory extends ActionFactory<IndexAction, ExecutableInd
@Override
public ExecutableIndexAction createExecutable(IndexAction action) {
return new ExecutableIndexAction(action, actionLogger, client, defaultTimeout, indexNamesParser);
return new ExecutableIndexAction(action, actionLogger, client, defaultTimeout);
}
}

12
watcher/src/main/java/org/elasticsearch/watcher/input/search/ExecutableSearchInput.java
View File

@ -18,7 +18,6 @@ import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.watcher.execution.WatchExecutionContext;
import org.elasticsearch.watcher.input.ExecutableInput;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.WatcherUtils;
import org.elasticsearch.watcher.support.XContentFilterKeysUtils;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
@ -37,24 +36,17 @@ public class ExecutableSearchInput extends ExecutableInput<SearchInput, SearchIn
private final ClientProxy client;
private final @Nullable TimeValue timeout;
private final @Nullable DynamicIndexName[] indexNames;
public ExecutableSearchInput(SearchInput input, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout, DynamicIndexName.Parser indexNameParser) {
public ExecutableSearchInput(SearchInput input, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout) {
super(input, logger);
this.client = client;
this.timeout = input.getTimeout() != null ? input.getTimeout() : defaultTimeout;
String[] indices = input.getSearchRequest().indices();
indexNames = indices != null ? indexNameParser.parse(indices, input.getDynamicNameTimeZone()) : null;
}
DynamicIndexName[] indexNames() {
return indexNames;
}
public SearchInput.Result execute(WatchExecutionContext ctx) {
SearchRequest request = null;
try {
request = WatcherUtils.createSearchRequestFromPrototype(input.getSearchRequest(), indexNames, ctx, null);
request = WatcherUtils.createSearchRequestFromPrototype(input.getSearchRequest(), ctx, null);
return doExecute(ctx, request);
} catch (Exception e) {
logger.error("failed to execute [{}] input for [{}]", e, SearchInput.TYPE, ctx.watch());

5
watcher/src/main/java/org/elasticsearch/watcher/input/search/SearchInputFactory.java
View File

@ -12,7 +12,6 @@ import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.watcher.input.InputFactory;
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
import java.io.IOException;
@ -23,14 +22,12 @@ import java.io.IOException;
public class SearchInputFactory extends InputFactory<SearchInput, SearchInput.Result, ExecutableSearchInput> {
private final ClientProxy client;
private final DynamicIndexName.Parser indexNameParser;
private final TimeValue defaultTimeout;
@Inject
public SearchInputFactory(Settings settings, ClientProxy client) {
super(Loggers.getLogger(ExecutableSimpleInput.class, settings));
this.client = client;
this.indexNameParser = new DynamicIndexName.Parser(settings, "watcher.input.search");
this.defaultTimeout = settings.getAsTime("watcher.input.search.default_timeout", null);
}
@ -46,6 +43,6 @@ public class SearchInputFactory extends InputFactory<SearchInput, SearchInput.Re
@Override
public ExecutableSearchInput createExecutable(SearchInput input) {
return new ExecutableSearchInput(input, inputLogger, client, defaultTimeout, indexNameParser);
return new ExecutableSearchInput(input, inputLogger, client, defaultTimeout);
}
}

339
watcher/src/main/java/org/elasticsearch/watcher/support/DynamicIndexName.java
View File

@ -1,339 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.watcher.support;
import org.elasticsearch.ElasticsearchParseException;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.joda.DateMathParser;
import org.elasticsearch.common.joda.FormatDateTimeFormatter;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.concurrent.Callable;
/**
*
*/
public class DynamicIndexName implements ToXContent {
public static final String DEFAULT_DATE_FORMAT = "YYYY.MM.dd";
private static final String EXPRESSION_LEFT_BOUND = "<";
private static final String EXPRESSION_RIGHT_BOUND = ">";
private static final char LEFT_BOUND = '{';
private static final char RIGHT_BOUND = '}';
private static final char ESCAPE_CHAR = '\\';
private final String text;
private final Expression expression;
private DynamicIndexName(String text, Expression expression) {
this.text = text;
this.expression = expression;
}
public String text() {
return text;
}
public String name(DateTime now) {
return expression.eval(now);
}
public static String[] names(DynamicIndexName[] indexNames, DateTime now) {
String[] names = new String[indexNames.length];
for (int i = 0; i < names.length; i++) {
names[i] = indexNames[i].name(now);
}
return names;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
DynamicIndexName that = (DynamicIndexName) o;
return text.equals(that.text);
}
@Override
public int hashCode() {
return text.hashCode();
}
@Override
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
return builder.value(text);
}
public static String defaultDateFormat(Settings settings, String componentPrefix) {
if (componentPrefix == null) {
return defaultDateFormat(settings);
}
return settings.get(componentPrefix + ".dynamic_indices.default_date_format", defaultDateFormat(settings));
}
public static String defaultDateFormat(Settings settings) {
return settings.get("watcher.dynamic_indices.default_date_format", DEFAULT_DATE_FORMAT);
}
public static DateTimeZone timeZone(Settings settings, String componentPrefix) {
if (componentPrefix == null) {
return timeZone(settings);
}
String timeZoneId = settings.get(componentPrefix + ".dynamic_indices.time_zone", DateTimeZone.UTC.getID());
return DateTimeZone.forID(timeZoneId);
}
public static DateTimeZone timeZone(Settings settings) {
String timeZoneId = settings.get("watcher.dynamic_indices.time_zone", DateTimeZone.UTC.getID());
return DateTimeZone.forID(timeZoneId);
}
interface Expression {
String eval(DateTime now);
}
static class StaticExpression implements Expression {
private final String value;
public StaticExpression(String value) {
this.value = value;
}
@Override
public String eval(DateTime now) {
return value;
}
}
static class DateMathExpression implements Expression {
private final DateMathParser dateMathParser;
private final String mathExpression;
private final FormatDateTimeFormatter formatter;
private final DateTimeZone timeZone;
public DateMathExpression(String defaultFormat, DateTimeZone timeZone, String expression) {
this.timeZone = timeZone;
int i = expression.indexOf(LEFT_BOUND);
String format;
if (i < 0) {
mathExpression = expression;
format = defaultFormat;
} else {
if (expression.lastIndexOf(RIGHT_BOUND) != expression.length() - 1) {
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. missing closing `}` for date math format", expression);
}
if (i == expression.length() - 2) {
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. missing date format", expression);
}
mathExpression = expression.substring(0, i);
format = expression.substring(i + 1, expression.length() - 1);
}
DateTimeFormatter parser = DateTimeFormat.forPattern(format).withZone(timeZone);
formatter = new FormatDateTimeFormatter(defaultFormat, parser, Locale.ROOT);
dateMathParser = new DateMathParser(formatter);
}
@Override
public String eval(final DateTime now) {
long millis = dateMathParser.parse(mathExpression, new Callable<Long>() {
@Override
public Long call() throws Exception {
return now.getMillis();
}
}, false, timeZone);
return formatter.printer().print(millis);
}
}
static class CompoundExpression implements Expression {
private final Expression[] parts;
public CompoundExpression(Expression[] parts) {
this.parts = parts;
}
@Override
public String eval(DateTime now) {
StringBuilder sb = new StringBuilder();
for (Expression part : parts) {
sb.append(part.eval(now));
}
return sb.toString();
}
static Expression parse(String defaultDateFormat, DateTimeZone timeZone, char[] text, int from, int length) {
boolean dynamic = false;
List<Expression> expressions = new ArrayList<>();
StringBuilder sb = new StringBuilder();
boolean inPlaceHolder = false;
boolean inDateFormat = false;
boolean escape = false;
for (int i = from; i < length; i++) {
boolean escapedChar = escape;
if (escape) {
escape = false;
}
char c = text[i];
if (c == ESCAPE_CHAR) {
if (escapedChar) {
sb.append(c);
escape = false;
} else {
escape = true;
}
continue;
}
if (inPlaceHolder) {
switch (c) {
case LEFT_BOUND:
if (inDateFormat && escapedChar) {
sb.append(c);
} else if (!inDateFormat) {
inDateFormat = true;
sb.append(c);
} else {
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. invalid character in placeholder at position [{}]", new String(text, from, length), i);
}
break;
case RIGHT_BOUND:
if (inDateFormat && escapedChar) {
sb.append(c);
} else if (inDateFormat) {
inDateFormat = false;
sb.append(c);
} else {
expressions.add(new DateMathExpression(defaultDateFormat, timeZone, sb.toString()));
sb = new StringBuilder();
inPlaceHolder = false;
dynamic = true;
}
break;
default:
sb.append(c);
}
} else {
switch (c) {
case LEFT_BOUND:
if (escapedChar) {
sb.append(c);
} else {
expressions.add(new StaticExpression(sb.toString()));
sb = new StringBuilder();
inPlaceHolder = true;
}
break;
case RIGHT_BOUND:
if (!escapedChar) {
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. invalid character at position [{}]. " +
"`{` and `}` are reserved characters and should be escaped when used as part of the index name using `\\` (e.g. `\\{text\\}`)", new String(text, from, length), i);
}
sb.append(c);
break;
default:
sb.append(c);
}
}
}
if (inPlaceHolder) {
throw new ElasticsearchParseException("invalid dynamic name expression [{}]. date math placeholder is open ended", new String(text, from, length));
}
if (sb.length() > 0) {
expressions.add(new StaticExpression(sb.toString()));
}
if (!dynamic) {
// if all the expressions are static... lets optimize to a single static expression
sb = new StringBuilder();
for (Expression expression : expressions) {
sb.append(((StaticExpression) expression).value);
}
return new StaticExpression(sb.toString());
}
if (expressions.size() == 1) {
return expressions.get(0);
}
return new CompoundExpression(expressions.toArray(new Expression[expressions.size()]));
}
}
public static class Parser {
private final String defaultDateFormat;
private final DateTimeZone timeZone;
public Parser() {
this(DEFAULT_DATE_FORMAT, DateTimeZone.UTC);
}
public Parser(String defaultDateFormat, DateTimeZone timeZone) {
this.defaultDateFormat = defaultDateFormat;
this.timeZone = timeZone;
}
public Parser(Settings settings, String componentPrefix) {
this(defaultDateFormat(settings, componentPrefix), timeZone(settings, componentPrefix));
}
public DynamicIndexName parse(String template) {
return parse(template, null);
}
public DynamicIndexName parse(String template, @Nullable DateTimeZone timeZone) {
DateTimeZone tz = timeZone != null ? timeZone : this.timeZone;
if (template == null) {
return null;
}
if (!template.startsWith(EXPRESSION_LEFT_BOUND) || !template.endsWith(EXPRESSION_RIGHT_BOUND)) {
return new DynamicIndexName(template, new StaticExpression(template));
}
return new DynamicIndexName(template, CompoundExpression.parse(defaultDateFormat, tz, template.toCharArray(), 1, template.length() - 1));
}
public DynamicIndexName[] parse(String[] templates) {
return parse(templates, null);
}
public DynamicIndexName[] parse(String[] templates, @Nullable DateTimeZone timeZone) {
if (templates.length == 0) {
return null;
}
DynamicIndexName[] dynamicIndexNames = new DynamicIndexName[templates.length];
for (int i = 0; i < dynamicIndexNames.length; i++) {
dynamicIndexNames[i] = parse(templates[i], timeZone);
}
return dynamicIndexNames;
}
}
}

9
watcher/src/main/java/org/elasticsearch/watcher/support/WatcherUtils.java
View File

@ -53,16 +53,11 @@ public final class WatcherUtils {
return XContentHelper.convertToMap(builder.bytes(), false).v2();
}
public static SearchRequest createSearchRequestFromPrototype(SearchRequest requestPrototype, @Nullable DynamicIndexName[] dynamicIndexNames, WatchExecutionContext ctx, Payload payload) throws IOException {
String[] indices = dynamicIndexNames == null ?
requestPrototype.indices() :
DynamicIndexName.names(dynamicIndexNames, ctx.executionTime());
public static SearchRequest createSearchRequestFromPrototype(SearchRequest requestPrototype, WatchExecutionContext ctx, Payload payload) throws IOException {
SearchRequest request = new SearchRequest(requestPrototype)
.indicesOptions(requestPrototype.indicesOptions())
.searchType(requestPrototype.searchType())
.indices(indices)
.indices(requestPrototype.indices())
.types(requestPrototype.types());
// TODO: Revise this search template conversion code once search templates in core have been refactored once ES 2.0 is released.

12
watcher/src/main/java/org/elasticsearch/watcher/transform/search/ExecutableSearchTransform.java
View File

@ -12,7 +12,6 @@ import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.logging.ESLogger;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.watcher.execution.WatchExecutionContext;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.WatcherUtils;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
import org.elasticsearch.watcher.transform.ExecutableTransform;
@ -27,25 +26,18 @@ public class ExecutableSearchTransform extends ExecutableTransform<SearchTransfo
protected final ClientProxy client;
protected final @Nullable TimeValue timeout;
private final @Nullable DynamicIndexName[] indexNames;
public ExecutableSearchTransform(SearchTransform transform, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout, DynamicIndexName.Parser indexNameParser) {
public ExecutableSearchTransform(SearchTransform transform, ESLogger logger, ClientProxy client, @Nullable TimeValue defaultTimeout) {
super(transform, logger);
this.client = client;
this.timeout = transform.getTimeout() != null ? transform.getTimeout() : defaultTimeout;
String[] indices = transform.getRequest().indices();
this.indexNames = indices != null ? indexNameParser.parse(indices, transform.getDynamicNameTimeZone()) : null;
}
DynamicIndexName[] indexNames() {
return indexNames;
}
@Override
public SearchTransform.Result execute(WatchExecutionContext ctx, Payload payload) {
SearchRequest request = null;
try {
request = WatcherUtils.createSearchRequestFromPrototype(transform.getRequest(), indexNames, ctx, payload);
request = WatcherUtils.createSearchRequestFromPrototype(transform.getRequest(), ctx, payload);
SearchResponse resp = client.search(request, timeout);
return new SearchTransform.Result(request, new Payload.XContent(resp));
} catch (Exception e) {

5
watcher/src/main/java/org/elasticsearch/watcher/transform/search/SearchTransformFactory.java
View File

@ -10,7 +10,6 @@ import org.elasticsearch.common.logging.Loggers;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
import org.elasticsearch.watcher.transform.TransformFactory;
@ -22,14 +21,12 @@ import java.io.IOException;
public class SearchTransformFactory extends TransformFactory<SearchTransform, SearchTransform.Result, ExecutableSearchTransform> {
protected final ClientProxy client;
protected final DynamicIndexName.Parser indexNameParser;
private final TimeValue defaultTimeout;
@Inject
public SearchTransformFactory(Settings settings, ClientProxy client) {
super(Loggers.getLogger(ExecutableSearchTransform.class, settings));
this.client = client;
this.indexNameParser = new DynamicIndexName.Parser(settings, "watcher.transform.search");
this.defaultTimeout = settings.getAsTime("watcher.transform.search.default_timeout", null);
}
@ -45,6 +42,6 @@ public class SearchTransformFactory extends TransformFactory<SearchTransform, Se
@Override
public ExecutableSearchTransform createExecutable(SearchTransform transform) {
return new ExecutableSearchTransform(transform, transformLogger, client, defaultTimeout, indexNameParser);
return new ExecutableSearchTransform(transform, transformLogger, client, defaultTimeout);
}
}

48
watcher/src/test/java/org/elasticsearch/watcher/actions/index/IndexActionTests.java
View File

@ -21,15 +21,12 @@ import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.watcher.actions.Action;
import org.elasticsearch.watcher.actions.Action.Result.Status;
import org.elasticsearch.watcher.execution.WatchExecutionContext;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.WatcherDateTimeUtils;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
import org.elasticsearch.watcher.support.xcontent.XContentSource;
import org.elasticsearch.watcher.test.WatcherTestUtils;
import org.elasticsearch.watcher.watch.Payload;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
import org.junit.Test;
import java.util.Arrays;
@ -73,7 +70,7 @@ public class IndexActionTests extends ESIntegTestCase {
}
IndexAction action = new IndexAction("test-index", "test-type", timestampField, null, null);
ExecutableIndexAction executable = new ExecutableIndexAction(action, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableIndexAction executable = new ExecutableIndexAction(action, logger, ClientProxy.of(client()), null);
DateTime executionTime = DateTime.now(UTC);
Payload payload = randomBoolean() ? new Payload.Simple("foo", "bar") : new Payload.Simple("_doc", ImmutableMap.of("foo", "bar"));
WatchExecutionContext ctx = WatcherTestUtils.mockExecutionContext("_id", executionTime, payload);
@ -136,7 +133,7 @@ public class IndexActionTests extends ESIntegTestCase {
);
IndexAction action = new IndexAction("test-index", "test-type", timestampField, null, null);
ExecutableIndexAction executable = new ExecutableIndexAction(action, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableIndexAction executable = new ExecutableIndexAction(action, logger, ClientProxy.of(client()), null);
DateTime executionTime = DateTime.now(UTC);
WatchExecutionContext ctx = WatcherTestUtils.mockExecutionContext("_id", executionTime, new Payload.Simple("_doc", list));
@ -217,47 +214,6 @@ public class IndexActionTests extends ESIntegTestCase {
assertThat(executable.action().timeout, equalTo(writeTimeout));
}
@Test
public void testParser_DynamicIndex() throws Exception {
DateTime now = DateTime.now(UTC);
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.forOffsetHours(-2) : null;
if (timeZone != null) {
now = now.withHourOfDay(0).withMinuteOfHour(0);
}
XContentBuilder builder = jsonBuilder();
builder.startObject()
.field(IndexAction.Field.INDEX.getPreferredName(), "<idx-{now/d}>")
.field(IndexAction.Field.DOC_TYPE.getPreferredName(), "test-type");
boolean timeZoneInWatch = randomBoolean();
if (timeZone != null && timeZoneInWatch) {
builder.field(IndexAction.Field.DYNAMIC_NAME_TIMEZONE.getPreferredName(), timeZone);
}
builder.endObject();
Settings.Builder settings = Settings.builder();
if (timeZone != null && !timeZoneInWatch) {
settings.put("watcher.actions.index.dynamic_indices.time_zone", timeZone);
}
IndexActionFactory actionParser = new IndexActionFactory(settings.build(), ClientProxy.of(client()));
XContentParser parser = JsonXContent.jsonXContent.createParser(builder.bytes());
parser.nextToken();
ExecutableIndexAction executable = actionParser.parseExecutable(randomAsciiOfLength(5), randomAsciiOfLength(3), parser);
assertThat(executable, notNullValue());
assertThat(executable.action().index, is("<idx-{now/d}>"));
String indexName = executable.indexName().name(now);
if (timeZone != null) {
now = now.withZone(timeZone);
}
assertThat(indexName, is("idx-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now)));
}
@Test
public void testParser_Failure() throws Exception {
XContentBuilder builder = jsonBuilder();

62
watcher/src/test/java/org/elasticsearch/watcher/input/search/SearchInputTests.java
View File

@ -7,7 +7,6 @@ package org.elasticsearch.watcher.input.search;
import com.google.common.collect.ImmutableMap;
import org.elasticsearch.ElasticsearchParseException;
import org.elasticsearch.action.indexedscripts.put.PutIndexedScriptRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchType;
@ -20,7 +19,6 @@ import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.common.xcontent.json.JsonXContent;
import org.elasticsearch.common.xcontent.support.XContentMapValues;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.sort.SortBuilders;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import org.elasticsearch.watcher.actions.ActionStatus;
@ -31,7 +29,6 @@ import org.elasticsearch.watcher.execution.TriggeredExecutionContext;
import org.elasticsearch.watcher.execution.WatchExecutionContext;
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
import org.elasticsearch.watcher.input.simple.SimpleInput;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
import org.elasticsearch.watcher.support.text.TextTemplate;
import org.elasticsearch.watcher.trigger.schedule.IntervalSchedule;
@ -41,8 +38,6 @@ import org.elasticsearch.watcher.watch.Payload;
import org.elasticsearch.watcher.watch.Watch;
import org.elasticsearch.watcher.watch.WatchStatus;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
import org.junit.Test;
import java.io.IOException;
@ -116,7 +111,7 @@ public class SearchInputTests extends ESIntegTestCase {
.request()
.source(searchSourceBuilder);
ExecutableSearchInput searchInput = new ExecutableSearchInput(new SearchInput(request, null, null, null), logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableSearchInput searchInput = new ExecutableSearchInput(new SearchInput(request, null, null, null), logger, ClientProxy.of(client()), null);
WatchExecutionContext ctx = new TriggeredExecutionContext(
new Watch("test-watch",
new ScheduleTrigger(new IntervalSchedule(new IntervalSchedule.Interval(1, IntervalSchedule.Interval.Unit.MINUTES))),
@ -223,7 +218,7 @@ public class SearchInputTests extends ESIntegTestCase {
.request()
.source(searchSourceBuilder);
ExecutableSearchInput searchInput = new ExecutableSearchInput(new SearchInput(request, null, null, null), logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableSearchInput searchInput = new ExecutableSearchInput(new SearchInput(request, null, null, null), logger, ClientProxy.of(client()), null);
WatchExecutionContext ctx = new TriggeredExecutionContext(
new Watch("test-watch",
new ScheduleTrigger(new IntervalSchedule(new IntervalSchedule.Interval(1, IntervalSchedule.Interval.Unit.MINUTES))),
@ -266,57 +261,6 @@ public class SearchInputTests extends ESIntegTestCase {
assertThat(searchInput.getTimeout(), equalTo(timeout));
}
@Test
public void testParser_IndexNames() throws Exception {
SearchRequest request = client().prepareSearch()
.setSearchType(ExecutableSearchInput.DEFAULT_SEARCH_TYPE)
.setIndices("test", "<test-{now/d-1d}>")
.request()
.source(searchSource()
.query(boolQuery().must(matchQuery("event_type", "a")).filter(rangeQuery("_timestamp").from("{{ctx.trigger.scheduled_time}}||-30s").to("{{ctx.trigger.triggered_time}}"))));
DateTime now = DateTime.now(UTC);
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.forOffsetHours(-2) : null;
if (timeZone != null) {
now = now.withHourOfDay(0).withMinuteOfHour(0);
}
boolean timeZoneInWatch = randomBoolean();
SearchInput input = timeZone != null && timeZoneInWatch ?
new SearchInput(request, null, null, timeZone) :
new SearchInput(request, null, null, null);
XContentBuilder builder = jsonBuilder().value(input);
XContentParser parser = JsonXContent.jsonXContent.createParser(builder.bytes());
parser.nextToken();
String dateFormat;
Settings.Builder settingsBuilder = Settings.builder();
if (randomBoolean()) {
dateFormat = DynamicIndexName.DEFAULT_DATE_FORMAT;
} else {
dateFormat = "YYYY-MM-dd";
settingsBuilder.put("watcher.input.search.dynamic_indices.default_date_format", dateFormat);
}
if (timeZone != null && !timeZoneInWatch) {
settingsBuilder.put("watcher.input.search.dynamic_indices.time_zone", timeZone);
}
SearchInputFactory factory = new SearchInputFactory(settingsBuilder.build(), ClientProxy.of(client()));
ExecutableSearchInput executable = factory.parseExecutable("_id", parser);
DynamicIndexName[] indexNames = executable.indexNames();
assertThat(indexNames, notNullValue());
String[] names = DynamicIndexName.names(indexNames, now);
assertThat(names, notNullValue());
assertThat(names.length, is(2));
if (timeZone != null) {
now = now.withZone(timeZone);
}
assertThat(names, arrayContaining("test", "test-" + DateTimeFormat.forPattern(dateFormat).print(now.minusDays(1))));
}
private WatchExecutionContext createContext() {
return new TriggeredExecutionContext(
new Watch("test-watch",
@ -340,7 +284,7 @@ public class SearchInputTests extends ESIntegTestCase {
SearchInput si = siBuilder.build();
ExecutableSearchInput searchInput = new ExecutableSearchInput(si, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableSearchInput searchInput = new ExecutableSearchInput(si, logger, ClientProxy.of(client()), null);
return searchInput.execute(ctx);
}

12
watcher/src/test/java/org/elasticsearch/watcher/support/DynamicIndexNameIntegrationTests.java
View File

@ -83,11 +83,12 @@ public class DynamicIndexNameIntegrationTests extends AbstractWatcherIntegration
flush();
refresh();
String indexNameDateMathExpressions = "<idx-{now/d}>";
WatcherClient watcherClient = watcherClient();
PutWatchResponse putWatchResponse = watcherClient.preparePutWatch("_id")
.setSource(watchBuilder()
.trigger(schedule(interval(5, IntervalSchedule.Interval.Unit.SECONDS)))
.input(searchInput(new SearchRequest("<idx-{now/d}>").types("type"))))
.input(searchInput(new SearchRequest(indexNameDateMathExpressions).types("type"))))
.get();
assertThat(putWatchResponse.isCreated(), is(true));
@ -96,23 +97,24 @@ public class DynamicIndexNameIntegrationTests extends AbstractWatcherIntegration
flush();
refresh();
SearchResponse response = searchHistory(searchSource().query(matchQuery("result.input.search.request.indices", indexName)));
SearchResponse response = searchHistory(searchSource().query(matchQuery("result.input.search.request.indices", indexNameDateMathExpressions)));
assertThat(response.getHits().getTotalHits(), is(1L));
}
@Test
public void testDynamicIndexSearchTransform() throws Exception {
final String indexName = "idx-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(timeWarp().clock().nowUTC());
String indexName = "idx-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(timeWarp().clock().nowUTC());
createIndex(indexName);
index(indexName, "type", "1", "key", "value");
flush();
refresh();
final String indexNameDateMathExpressions = "<idx-{now/d}>";
WatcherClient watcherClient = watcherClient();
PutWatchResponse putWatchResponse = watcherClient.preparePutWatch("_id")
.setSource(watchBuilder()
.trigger(schedule(interval(5, IntervalSchedule.Interval.Unit.SECONDS)))
.transform(searchTransform(new SearchRequest("<idx-{now/d}>").types("type")))
.transform(searchTransform(new SearchRequest(indexNameDateMathExpressions).types("type")))
.addAction("log", loggingAction("heya")))
.get();
@ -125,7 +127,7 @@ public class DynamicIndexNameIntegrationTests extends AbstractWatcherIntegration
SearchResponse response = searchWatchRecords(new Callback<SearchRequestBuilder>() {
@Override
public void handle(SearchRequestBuilder builder) {
builder.setQuery(matchQuery("result.transform.search.request.indices", indexName));
builder.setQuery(matchQuery("result.transform.search.request.indices", indexNameDateMathExpressions));
}
});
assertThat(response.getHits().getTotalHits(), is(1L));

266
watcher/src/test/java/org/elasticsearch/watcher/support/DynamicIndexNameTests.java
View File

@ -1,266 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.watcher.support;
import org.elasticsearch.ElasticsearchParseException;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.test.ESTestCase;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
import org.junit.Test;
import static org.hamcrest.CoreMatchers.nullValue;
import static org.hamcrest.Matchers.*;
/**
*
*/
public class DynamicIndexNameTests extends ESTestCase {
@Test
public void testNormal() throws Exception {
String indexName = randomAsciiOfLength(10);
DateTime now = DateTime.now(DateTimeZone.UTC);
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName indexNames = parser.parse(indexName);
String name = indexNames.name(now);
assertThat(name, equalTo(indexName));
}
@Test
public void testExpression() throws Exception {
DateTime now = DateTime.now(DateTimeZone.UTC);
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName indexNames = parser.parse("<.marvel-{now}>");
String name = indexNames.name(now);
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now)));
}
@Test
public void testNullOrEmpty() throws Exception {
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName indexName = parser.parse((String) null);
assertThat(indexName, nullValue());
DynamicIndexName[] indexNames = parser.parse(Strings.EMPTY_ARRAY);
assertThat(indexNames, nullValue());
}
@Test
public void testExpression_Static() throws Exception {
DateTime now = DateTime.now(DateTimeZone.UTC);
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName indexNames = parser.parse("<.marvel-test>");
String name = indexNames.name(now);
assertThat(name, equalTo(".marvel-test"));
}
@Test
public void testExpression_MultiParts() throws Exception {
DateTime now = DateTime.now(DateTimeZone.UTC);
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName indexNames = parser.parse("<.text1-{now/d}-text2-{now/M}>");
String name = indexNames.name(now);
assertThat(name, equalTo(".text1-"
+ DateTimeFormat.forPattern("YYYY.MM.dd").print(now)
+ "-text2-"
+ DateTimeFormat.forPattern("YYYY.MM.dd").print(now.withDayOfMonth(1))));
}
@Test
public void testExpression_CustomFormat() throws Exception {
DateTime now = DateTime.now(DateTimeZone.UTC);
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName indexNames = parser.parse("<.marvel-{now/d{YYYY.MM.dd}}>");
String name = indexNames.name(now);
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now)));
}
@Test
public void testExpression_CustomTimeZone() throws Exception {
DateTimeZone timeZone;
int hoursOffset;
int minutesOffset = 0;
if (randomBoolean()) {
hoursOffset = randomIntBetween(-12, 14);
timeZone = DateTimeZone.forOffsetHours(hoursOffset);
} else {
hoursOffset = randomIntBetween(-11, 13);
minutesOffset = randomIntBetween(0, 59);
timeZone = DateTimeZone.forOffsetHoursMinutes(hoursOffset, minutesOffset);
}
DateTime now;
if (hoursOffset >= 0) {
// rounding to next day 00:00
now = DateTime.now(DateTimeZone.UTC).plusHours(hoursOffset).plusMinutes(minutesOffset).withHourOfDay(0).withMinuteOfHour(0).withSecondOfMinute(0);
} else {
// rounding to today 00:00
now = DateTime.now(DateTimeZone.UTC).withHourOfDay(0).withMinuteOfHour(0).withSecondOfMinute(0);
}
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", timeZone);
DynamicIndexName indexNames = parser.parse("<.marvel-{now/d{YYYY.MM.dd}}>");
String name = indexNames.name(now);
logger.info("timezone: [{}], now [{}], name: [{}]", timeZone, now, name);
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.withZone(timeZone))));
}
@Test
public void testExpression_CustomTimeZone_OnParse() throws Exception {
DateTimeZone timeZone;
int hoursOffset;
int minutesOffset = 0;
if (randomBoolean()) {
hoursOffset = randomIntBetween(-12, 14);
timeZone = DateTimeZone.forOffsetHours(hoursOffset);
} else {
hoursOffset = randomIntBetween(-11, 13);
minutesOffset = randomIntBetween(0, 59);
timeZone = DateTimeZone.forOffsetHoursMinutes(hoursOffset, minutesOffset);
}
DateTime now;
if (hoursOffset >= 0) {
// rounding to next day 00:00
now = DateTime.now(DateTimeZone.UTC).plusHours(hoursOffset).plusMinutes(minutesOffset).withHourOfDay(0).withMinuteOfHour(0).withSecondOfMinute(0);
} else {
// rounding to today 00:00
now = DateTime.now(DateTimeZone.UTC).withHourOfDay(0).withMinuteOfHour(0).withSecondOfMinute(0);
}
Settings settings = Settings.builder()
.put("watcher.dynamic_indices.default_date_format", "YYYY.MM.dd")
.put("watcher.dynamic_indices.time_zone", "-12")
.put("watcher.foo.dynamic_indices.time_zone", "-12")
.build();
DynamicIndexName.Parser parser = new DynamicIndexName.Parser(settings, "watcher.foo");
DynamicIndexName indexNames = parser.parse("<.marvel-{now/d{YYYY.MM.dd}}>", timeZone);
String name = indexNames.name(now);
logger.info("timezone: [{}], now [{}], name: [{}]", timeZone, now, name);
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.withZone(timeZone))));
}
@Test
public void testExpression_EscapeStatic() throws Exception {
DateTime now = DateTime.now(DateTimeZone.UTC);
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName indexNames = parser.parse("<.mar\\{v\\}el-{now/d}>");
String name = indexNames.name(now);
assertThat(name, equalTo(".mar{v}el-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now)));
}
@Test
public void testExpression_EscapeDateFormat() throws Exception {
DateTime now = DateTime.now(DateTimeZone.UTC);
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName indexNames = parser.parse("<.marvel-{now/d{'\\{year\\}'YYYY}}>");
String name = indexNames.name(now);
assertThat(name, equalTo(".marvel-" + DateTimeFormat.forPattern("'{year}'YYYY").print(now)));
}
@Test
public void testExpression_MixedArray() throws Exception {
DateTime now = DateTime.now(DateTimeZone.UTC);
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
DynamicIndexName[] indexNames = parser.parse(new String[] {
"name1",
"<.marvel-{now/d}>",
"name2",
"<.logstash-{now/M{YYYY.MM}}>"
});
String[] names = new String[indexNames.length];
for (int i = 0; i < names.length; i++) {
names[i] = indexNames[i].name(now);
}
assertThat(names.length, is(4));
assertThat(names, arrayContaining(
"name1",
".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now),
"name2",
".logstash-" + DateTimeFormat.forPattern("YYYY.MM").print(now.withDayOfMonth(1))));
}
@Test(expected = ElasticsearchParseException.class)
public void testExpression_Invalid_Unescaped() throws Exception {
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
parser.parse("<.mar}vel-{now/d}>");
}
@Test(expected = ElasticsearchParseException.class)
public void testExpression_Invalid_DateMathFormat() throws Exception {
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
parser.parse("<.marvel-{now/d{}>");
}
@Test(expected = ElasticsearchParseException.class)
public void testExpression_Invalid_EmptyDateMathFormat() throws Exception {
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
parser.parse("<.marvel-{now/d{}}>");
}
@Test(expected = ElasticsearchParseException.class)
public void testExpression_Invalid_OpenEnded() throws Exception {
DynamicIndexName.Parser parser = new DynamicIndexName.Parser("YYYY.MM.dd", DateTimeZone.UTC);
parser.parse("<.marvel-{now/d>");
}
@Test
public void testDefaultDateFormat_Default() throws Exception {
String dateFormat = DynamicIndexName.defaultDateFormat(Settings.EMPTY);
assertThat(dateFormat, is("YYYY.MM.dd"));
}
@Test
public void testDefaultDateFormat() throws Exception {
Settings settings = Settings.builder()
.put("watcher.dynamic_indices.default_date_format", "YYYY.MM")
.build();
String dateFormat = randomBoolean() ?
DynamicIndexName.defaultDateFormat(settings) :
DynamicIndexName.defaultDateFormat(settings, null);
assertThat(dateFormat, is("YYYY.MM"));
}
@Test
public void testDefaultDateFormat_Component() throws Exception {
Settings settings = Settings.builder()
.put("watcher.dynamic_indices.default_date_format", "YYYY.MM")
.put("watcher.foo.dynamic_indices.default_date_format", "YYY.MM")
.build();
String dateFormat = DynamicIndexName.defaultDateFormat(settings, "watcher.foo");
assertThat(dateFormat, is("YYY.MM"));
}
@Test
public void testTimeZone_Default() throws Exception {
DateTimeZone timeZone = DynamicIndexName.timeZone(Settings.EMPTY);
assertThat(timeZone, is(DateTimeZone.UTC));
}
@Test
public void testTimeZone() throws Exception {
DateTimeZone timeZone = DateTimeZone.forOffsetHours(randomIntBetween(-12, 14));
Settings settings = Settings.builder()
.put("watcher.dynamic_indices.time_zone", timeZone)
.build();
DateTimeZone resolvedTimeZone = randomBoolean() ?
DynamicIndexName.timeZone(settings) :
DynamicIndexName.timeZone(settings, null);
assertThat(timeZone, is(resolvedTimeZone));
}
@Test
public void testTimeZone_Component() throws Exception {
DateTimeZone timeZone = DateTimeZone.forOffsetHours(randomIntBetween(-11, 14));
Settings settings = Settings.builder()
.put("watcher.dynamic_indices.time_zone", "-12")
.put("watcher.foo.dynamic_indices.time_zone", timeZone)
.build();
DateTimeZone resolvedTimeZone = DynamicIndexName.timeZone(settings, "watcher.foo");
assertThat(resolvedTimeZone, is(timeZone));
}
}

4
watcher/src/test/java/org/elasticsearch/watcher/test/WatcherTestUtils.java
View File

@ -41,7 +41,6 @@ import org.elasticsearch.watcher.input.search.ExecutableSearchInput;
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
import org.elasticsearch.watcher.input.simple.SimpleInput;
import org.elasticsearch.watcher.license.LicenseService;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.Script;
import org.elasticsearch.watcher.support.WatcherUtils;
import org.elasticsearch.watcher.support.http.HttpClient;
@ -69,7 +68,6 @@ import org.joda.time.DateTime;
import javax.mail.internet.AddressException;
import java.io.IOException;
import java.lang.reflect.Constructor;
import java.nio.charset.StandardCharsets;
import java.util.*;
@ -222,7 +220,7 @@ public final class WatcherTestUtils {
new ScheduleTrigger(new CronSchedule("0/5 * * * * ? *")),
new ExecutableSimpleInput(new SimpleInput(new Payload.Simple(inputData)), logger),
new ExecutableScriptCondition(new ScriptCondition(Script.inline("return true").build()), logger, scriptService),
new ExecutableSearchTransform(new SearchTransform(transformRequest, null, null), logger, client, null, new DynamicIndexName.Parser()),
new ExecutableSearchTransform(new SearchTransform(transformRequest, null, null), logger, client, null),
new TimeValue(0),
new ExecutableActions(actions),
metadata,

73
watcher/src/test/java/org/elasticsearch/watcher/transform/search/SearchTransformTests.java
View File

@ -30,7 +30,6 @@ import org.elasticsearch.watcher.execution.TriggeredExecutionContext;
import org.elasticsearch.watcher.execution.WatchExecutionContext;
import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
import org.elasticsearch.watcher.input.simple.SimpleInput;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.init.proxy.ClientProxy;
import org.elasticsearch.watcher.support.text.TextTemplate;
import org.elasticsearch.watcher.transform.Transform;
@ -42,8 +41,6 @@ import org.elasticsearch.watcher.watch.Payload;
import org.elasticsearch.watcher.watch.Watch;
import org.elasticsearch.watcher.watch.WatchStatus;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
import org.junit.Test;
import java.io.IOException;
@ -128,7 +125,7 @@ public class SearchTransformTests extends ESIntegTestCase {
.endObject()
.endObject().bytes());
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null);
WatchExecutionContext ctx = mockExecutionContext("_name", EMPTY_PAYLOAD);
@ -165,7 +162,7 @@ public class SearchTransformTests extends ESIntegTestCase {
.endObject()
.endObject().bytes());
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null);
WatchExecutionContext ctx = mockExecutionContext("_name", EMPTY_PAYLOAD);
@ -211,7 +208,7 @@ public class SearchTransformTests extends ESIntegTestCase {
.must(termQuery("value", "{{ctx.payload.value}}"))));
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableSearchTransform transform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null);
ScheduleTriggerEvent event = new ScheduleTriggerEvent("_name", parseDate("2015-01-04T00:00:00", UTC), parseDate("2015-01-01T00:00:00", UTC));
WatchExecutionContext ctx = mockExecutionContext("_name", parseDate("2015-01-04T00:00:00", UTC), event, EMPTY_PAYLOAD);
@ -303,68 +300,6 @@ public class SearchTransformTests extends ESIntegTestCase {
assertThat(executable.transform().getTimeout(), equalTo(readTimeout));
}
@Test
public void testParser_WithIndexNames() throws Exception {
SearchType searchType = getRandomSupportedSearchType();
XContentBuilder builder = jsonBuilder().startObject();
builder.startObject("request");
builder.array("indices", "idx", "<idx-{now/d-3d}>");
if (searchType != null) {
builder.field("search_type", searchType.name());
}
DateTime now = DateTime.now(UTC);
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.forOffsetHours(-2) : null;
if (timeZone != null) {
now = now.withHourOfDay(0).withMinuteOfHour(0);
}
builder.startObject("body")
.startObject("query")
.startObject("match_all")
.endObject()
.endObject()
.endObject();
builder.endObject();
boolean timeZoneInWatch = randomBoolean();
if (timeZone != null && timeZoneInWatch) {
builder.field(SearchTransform.Field.DYNAMIC_NAME_TIMEZONE.getPreferredName(), timeZone);
}
builder.endObject();
XContentParser parser = JsonXContent.jsonXContent.createParser(builder.bytes());
parser.nextToken();
String dateFormat;
Settings.Builder settingsBuilder = Settings.builder();
if (randomBoolean()) {
dateFormat = DynamicIndexName.DEFAULT_DATE_FORMAT;
} else {
dateFormat = "YYYY-MM";
settingsBuilder.put("watcher.transform.search.dynamic_indices.default_date_format", dateFormat);
}
if (timeZone != null && !timeZoneInWatch) {
settingsBuilder.put("watcher.transform.search.dynamic_indices.time_zone", timeZone);
}
Settings settings = settingsBuilder.build();
SearchTransformFactory transformFactory = new SearchTransformFactory(settings, ClientProxy.of(client()));
ExecutableSearchTransform executable = transformFactory.parseExecutable("_id", parser);
DynamicIndexName[] indexNames = executable.indexNames();
assertThat(indexNames, notNullValue());
String[] names = DynamicIndexName.names(indexNames, now);
assertThat(names, notNullValue());
assertThat(names.length, is(2));
if (timeZone != null) {
now = now.withZone(timeZone);
}
assertThat(names, arrayContaining("idx", "idx-" + DateTimeFormat.forPattern(dateFormat).print(now.minusDays(3))));
}
@Test
public void testSearch_InlineTemplate() throws Exception {
WatchExecutionContext ctx = createContext();
@ -501,7 +436,7 @@ public class SearchTransformTests extends ESIntegTestCase {
ensureGreen("test-search-index");
SearchTransform searchTransform = TransformBuilders.searchTransform(request).build();
ExecutableSearchTransform executableSearchTransform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null, new DynamicIndexName.Parser());
ExecutableSearchTransform executableSearchTransform = new ExecutableSearchTransform(searchTransform, logger, ClientProxy.of(client()), null);
return executableSearchTransform.execute(ctx, Payload.Simple.EMPTY);
}

11
watcher/src/test/java/org/elasticsearch/watcher/watch/WatchTests.java
View File

@ -60,7 +60,6 @@ import org.elasticsearch.watcher.input.simple.ExecutableSimpleInput;
import org.elasticsearch.watcher.input.simple.SimpleInput;
import org.elasticsearch.watcher.input.simple.SimpleInputFactory;
import org.elasticsearch.watcher.license.LicenseService;
import org.elasticsearch.watcher.support.DynamicIndexName;
import org.elasticsearch.watcher.support.Script;
import org.elasticsearch.watcher.support.WatcherUtils;
import org.elasticsearch.watcher.support.clock.Clock;
@ -121,7 +120,6 @@ public class WatchTests extends ESTestCase {
private SecretService secretService;
private LicenseService licenseService;
private ESLogger logger;
private DynamicIndexName.Parser indexNameParser;
private Settings settings = Settings.EMPTY;
@Before
@ -136,7 +134,6 @@ public class WatchTests extends ESTestCase {
licenseService = mock(LicenseService.class);
authRegistry = new HttpAuthRegistry(ImmutableMap.of("basic", (HttpAuthFactory) new BasicAuthFactory(secretService)));
logger = Loggers.getLogger(WatchTests.class);
indexNameParser = new DynamicIndexName.Parser();
}
@Test
@ -311,7 +308,7 @@ public class WatchTests extends ESTestCase {
switch (type) {
case SearchInput.TYPE:
SearchInput searchInput = searchInput(WatcherTestUtils.newInputSearchRequest("idx")).build();
return new ExecutableSearchInput(searchInput, logger, client, null, indexNameParser);
return new ExecutableSearchInput(searchInput, logger, client, null);
default:
SimpleInput simpleInput = InputBuilders.simpleInput(ImmutableMap.<String, Object>builder().put("_key", "_val")).build();
return new ExecutableSimpleInput(simpleInput, logger);
@ -370,12 +367,12 @@ public class WatchTests extends ESTestCase {
case ScriptTransform.TYPE:
return new ExecutableScriptTransform(new ScriptTransform(Script.inline("_script").build()), logger, scriptService);
case SearchTransform.TYPE:
return new ExecutableSearchTransform(new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), logger, client, null, indexNameParser);
return new ExecutableSearchTransform(new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), logger, client, null);
default: // chain
ChainTransform chainTransform = new ChainTransform(Arrays.asList(
new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), new ScriptTransform(Script.inline("_script").build())));
return new ExecutableChainTransform(chainTransform, logger, Arrays.<ExecutableTransform>asList(
new ExecutableSearchTransform(new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), logger, client, null, indexNameParser),
new ExecutableSearchTransform(new SearchTransform(matchAllRequest(WatcherUtils.DEFAULT_INDICES_OPTIONS), timeout, timeZone), logger, client, null),
new ExecutableScriptTransform(new ScriptTransform(Script.inline("_script").build()), logger, scriptService)));
}
}
@ -402,7 +399,7 @@ public class WatchTests extends ESTestCase {
DateTimeZone timeZone = randomBoolean() ? DateTimeZone.UTC : null;
TimeValue timeout = randomBoolean() ? TimeValue.timeValueSeconds(30) : null;
IndexAction action = new IndexAction("_index", "_type", null, timeout, timeZone);
list.add(new ActionWrapper("_index_" + randomAsciiOfLength(8), randomThrottler(), randomTransform(), new ExecutableIndexAction(action, logger, client, null, indexNameParser)));
list.add(new ActionWrapper("_index_" + randomAsciiOfLength(8), randomThrottler(), randomTransform(), new ExecutableIndexAction(action, logger, client, null)));
}
if (randomBoolean()) {
HttpRequestTemplate httpRequest = HttpRequestTemplate.builder("test.host", randomIntBetween(8000, 9000))