Today `DynamicIndexName` generates the names based on UTC. This doesn't work for environments where the indices rollover is based on a local/different timezone.
The timezone can be configured:
* globally with the `watcher.dynamic_indices.time_zone` setting
* collectively to all `search` input, `search` transform and `index` action with `watcher.input.search.dynamic_indices.time_zone`, `watcher.transform.search.dynamic_indices.time_zone` and `watcher.actions.index.dynamic_indices.time_zone` respectively
* locally on the watch as part of the `search` input, `search` transform and/or `index` action definition.
This commit revealed a set of bugs in the `search` input, `search` transform and/or `index` action when it comes to xcontent serialization. In short, when the user didn't define a timeout, a default timeout was set on them, which meant their serialized xcontent did not match the original one (from which they're deserialized). This bug also meant that the global default client timeouts (defined in the `ClientProxy` never took effect.
This issue revealed the bugs above as the `WatchTests` were enhanced to execute more settings permutations
Closeselastic/elasticsearch#632
Original commit: elastic/x-pack-elasticsearch@35a2191828
The TemplateRegistry adds templates based on if these index templates exist in the cluster state. Components that rely on index templates register their template config with the TemplateRegistry. The TemplateRegistry adds these templates in the background when a cluster state update occurs and add component index settings to the index template. Also when component index settings change, the index template will be updated by the TemplateRegistry.
If a registered index template gets deleted, it will be added back by the TemplateRegistry in background if in a cluster state the index template is missing.
Original commit: elastic/x-pack-elasticsearch@97f4f42160
was renamed to `_status` so it'll distinct from the other "formal" fields of the watch and also be aligned with the `_status` field that is returned by the Get API.
Also updated the installation docs
Original commit: elastic/x-pack-elasticsearch@1081b75d13
Added default timeout setting:
* `watcher.action.index.default_timeout` sets the timeout for `index` action, if no timeout is defined in the `index` action itself.
* `watcher.input.search.default_timeout` sets the timeout for `search` input, if no timeout is defined in the `search` input itself.
* `watcher.transform.search.default_timeout` sets the timeout for `search` transform, if no timeout is defined in the `search` transform itself.
Added general Watcher wide default timeout settings to the operations exposed in the client proxy:
* `watcher.internal.ops.search.default_timeout` for search related operations.
* `watcher.internal.ops.index.default_timeout` for index operations.
* `watcher.internal.ops.bulk.default_timeout` for bulk operations.
Original commit: elastic/x-pack-elasticsearch@5a3ef35a9d
The status isn't maintained by the user but rather by Watcher itself. The idea here is that the get watch api should return the watch as was provided to Watcher via the put watch api. The status will be reported under the top level `_status` field.
Original commit: elastic/x-pack-elasticsearch@54e2452493
At the moment if the starting fails watcher will hang in the `starting` phase. This is bad because any subsequent start will be ignored, because the the watcher state isn't `stopped`.
Original commit: elastic/x-pack-elasticsearch@5cbc1d2a13
The search input/transform rely on users configuring the search requests. Sometimes (often), these search requests are executed on time-based indices. The problem the
Until now, there's no way to define dynamic index names that are bound to time, which forces the request to search all the indices (for example, the Marvel watches se
This commit adds dynamic index name resolution. It works in the following way:
- and index name can be a simple string (indicating the static/absolute index name you're searching, incl. wildcards)
- an index name can also be a template. The template is enclosed within `<` and `>` (these are officially illegal characters for index names, so these are safe to use)
- the template can have both static parts to it and place holder parts. The place holders are enclosed within `{` and `}`. The place holder holds `date math` expression
* `"<.marvel-{now}>"` will resolve to `".marvel-2022.03.03"` (the default date format is `YYYY.MM.dd`)
* `"<.marvel-{now/M}>"` will resolve to `".marvel-2022.03.01"`
* `"<.marvel-{now{YYYY.MM}}>"` will resolve to `".marvel-2022.03"` (this one has a custom date format - `YYYY.MM`)
* `"<.marvel-{now/M-1M{YYYY.MM}}>"` will resolve to `".marvel-2022.02"`
The following is an example of a search input that searches marvel indices for the last 3 days (relies on the default Marvel indices format - `.marvel-YYYY.MM.dd`):
```
{
...
"input" : {
"search" : {
"request" : {
"indices" : [
"<.marvel-{now/d-2d}>",
"<.marvel-{now/d-1d}>",
"<.marvel-{now/d}>"
],
...
}
}
}
...
}
```
- `index` action was also updated to work with a dynamic index name (e.g. it's possible to index into daily indices by setting the index name to `<idx-{now}>`)
Original commit: elastic/x-pack-elasticsearch@9c15a96029
This changes the mappings for the audit indices to use doc_values for all fields
other than the request_body, which will have a lot of variance. Additionally, the
request_body field is no longer indexed.
Closeselastic/elasticsearch#918
Original commit: elastic/x-pack-elasticsearch@4917529ffa
The index audit trail is currently using a BulkProcessor directly, which under
certain conditions can result in a deadlock. This occurs when the BulkProcessor
is executing a bulk request that triggers another request on the same node and
a flush of the BulkProcessor is also triggered at the same time. The flush
operation holds the lock on the bulk processor but block on acquiring a permit
from the semaphore. The request that was triggered by the bulk request blocks
the release of the semaphore permit since it needs to add a new audit message
to the BulkProcessor.
This commit works around this issue by making use of a bounded queue between the
index audit trail and the BulkProcessor with a consumer thread that handles the
add calls to the BulkProcessor.
Additionally, a new state, INITIALIZED, was added for the lifecycle of the index
audit trail. This is needed for tests since the audit trail can stop, a new
cluster state update is received, and the ShieldLifecycleService will restart the
index audit trail. At the end of the tests, the test infrastructure interrupts all
the threads and this was causing tests to fail with a InterruptedException.
Finally, the test infrastructure was also deleting the template for the index audit
logs, so this commit adds the necessary logic to prevent the deletion of this
template.
Closeselastic/elasticsearch#920
Original commit: elastic/x-pack-elasticsearch@f1b0b47b99
The change fixes two bugs in the index audit trail implementation. The first is that
we did not always store the origin type with rest requests. The second is that a
conditional statement controlled the storage of the rest requests content, but the
conditional was based on a log level that had nothing to do with the index based
audit implementation.
Closeselastic/elasticsearch#932
Original commit: elastic/x-pack-elasticsearch@b309e261c3
we're not using the _timestamp field and the path option is no longer supported
in elasticsearch 2.0 so this commit removes the field from the mapping.
Original commit: elastic/x-pack-elasticsearch@399d835d1f
We need this as the `XContentSource` supports all xcontent constructs as the root construct, while xcontent in core only supports objects. For this reason, we can't rely on xcontent auto-detection of the xcontent type. We need to be explicit about it.
Original commit: elastic/x-pack-elasticsearch@a2ed944a21
This commit is a backwards compatibilty break for all watcher indices
that had a `throttel_period` set on their watches. `throttle_period` used
to be a numeric value but now is stored as a string AND requires a unit
like seconds or minutes etc. to prevent errors. All other time valiues like
http timeouts also require units now.
Closeselastic/elasticsearch#598
Original commit: elastic/x-pack-elasticsearch@e3b2c2a4af
- lowercase `beta` and `rc`
- replaced `.betaXXX` and `.rcXXXX` suffix with `-betaXXX` and `-rcXXX`
Original commit: elastic/x-pack-elasticsearch@843d01c647
This change renames the shield audit indices to keep naming consistent with other plugins.
The name of the index uses '_' to separate words, a '-' to separate the prefix from the time
portion, and '.'s to separate the different portions of the date.
Closeselastic/elasticsearch#925
Original commit: elastic/x-pack-elasticsearch@8ca6856e4a
Previously, we were just using the current time in milliseconds from the system
for dates and the indices were not being created for UTC dates. This change
uses UTC dates for timestamps and indices resolution for index auditing.
This also ensures that custom shield forbidden apis for tests are enforced.
Closeselastic/elasticsearch#916
Original commit: elastic/x-pack-elasticsearch@724d12cb7a
Robert Muir
Robert Muir
uboness
Martijn van Groningen
jaymode
Simon Willnauer