There may be current executions still going on during stopping.
We should try to wait for the current executions to have completed.
Otherwise we can run into a situation where we didn't delete the watch from the .triggered_watches index,
but did insert into the history index. Upon start this can lead to DocumentAlreadyExistsException,
because we already stored the history record during shutdown...
Introduced `CurrentExecutions` to handle this synchronization.
(we always first store the watch record and then remove the triggered watch)
Original commit: elastic/x-pack-elasticsearch@89c4a8d8ad
This commit adds support for indexing multiple documents with the `index` action. This is done by introducing a special `_doc` field. During action execution, the `_doc` field will be looked up in the payload. If found, the value of the field will be considered as the document that needs to be indexed. If the value is an array of objects, each object in that array will be treated as a separate document and all the documents in the array will be bulk indexed.
This commit also changes the result of the action to hold `XContentSource` rather than a payload (to avoid Map creation explosions). Th `XContentSource` was also extended to support lists.
Original commit: elastic/x-pack-elasticsearch@86f454b029
The `compare` condition result will now hold all resolved values, keyed by their associated parameter place holder strings.
Original commit: elastic/x-pack-elasticsearch@f930c77d54
Elasticsearch core no longer uses shaded dependencies by default, so this change makes
the plugin use the unshaded dependencies.
Original commit: elastic/x-pack-elasticsearch@495f636266
Elasticsearch core no longer uses shaded dependencies by default, so this change makes
the plugin use the unshaded dependencies.
Original commit: elastic/x-pack-elasticsearch@6f8b8b3feb
This change adds tests to ack a subset of a watch's actions, use a different throttle period per action in a watch, also adds tests to make sure that both the watch level and global level throttle_period are applied correctly.
Also updates the REST tests to make sure that throttle periods can be set at a watch and action level and are returned from the GET API.
Original commit: elastic/x-pack-elasticsearch@4b006c7830
Today, we simply return a SSLSocketFactory from a SSLContext, which provides
the default SSL configuration for sockets. This means that SSLv3 could still be
enabled on these sockets when running in an older JVM. This also means that
the ciphers and protocol settings are not honored for users of this socket factory,
which is currently the LDAP code.
This change returns a custom socket factory that delegates to the default socket
factory and sets the ciphers and protocols on the socket before returning the
socket.
Original commit: elastic/x-pack-elasticsearch@c4cfedfd51
Our current CA configuration creates certificates with only a CN and this caused
the regular expression in the PKI realm to fail. The default regular expression in
the PKI realm has been changed to allow for only a CN, while still maintaining the
ability to extract only the CN when other fields are present in the DN.
Additionally, the CA configuration has been updated so that is will copy any of the
originally specified fields (besides CN and EMAILADDRESS) over to the signed
certificate.
Original commit: elastic/x-pack-elasticsearch@ff27f69781
The IndicesTermsFilter Cache in core can leak data by not authorizing users prior to
retrieving data from the cache. We work around this by ensuring that the cache has
a maximum size of 0, effectively disabling it.
A test is also added to ensure that data is not leaked by this cache or the cache used by
the ScriptService in core.
Closeselastic/elasticsearch#854
Original commit: elastic/x-pack-elasticsearch@8a48bdad98
Today, we only have the option to require client authentication or not require it. A third case
exists where the server can want client authentication, but if the client does not have credentials
or wish to send credentials the connection will still be allowed.
This is beneficial for the PKI realm because it allows some clients to authenticate with PKI and
others to use another authentication token such as username and password authentication.
Closeselastic/elasticsearch#858
Original commit: elastic/x-pack-elasticsearch@e7cbc20bca
Until today it was required to provide a trigger event as part of the execute API. There are two issues with it:
1. It's not user friendly (you'd expect that the execute API would just work with just pointing to the watch)
2. The API could expose inconsistencies where on one hand it points to a watch (with a well defined trigger) on the other it enabled the user to provide completely different trigger (of a different type)
This change enable supporting default triggers by enabling the trigger engine to create a simulated trigger event. This enables the execute API to look up the trigger type of the pointed watch, and ask the trigger service to simulate an event for it. It is still possible to override the trigger event data by providing it via the `trigger_data` parameter in the API.
This simplifies the execute API and prepares it for future trigger types as well.
- This commit add missing integration tests for the execute API
- Also, removed unused `setIgnoreThrottling` from the execute request/builder.
Original commit: elastic/x-pack-elasticsearch@b494ae62e6
This change allows the httpinput to receive non json formatted data from a http endpoint
(such as the elasticsearch _cat apis). If non json is read it will be stored in the `payload._value` in the same
way that the `ScriptTransform` handles non map/json data returned by transforming scripts.
Added response_content_type to http input so that the expected content type in the response can be configured. This accepts `yaml`, `json` and `text` but will be overridden by the http headers.
Original commit: elastic/x-pack-elasticsearch@753d37f14e
Now that the watch record is write once and never read/parsed. We can de-normalize its format (the structure of the `watch_record` document) such it'd be optimal for searching and aggregating.
- `execution_result` renamed to `result` (for minimal verbosity)
- the structure of `trigger_event` changed to:
```
{
"trigger_event" : {
"type" : "<trigger_type>",
"triggered_time" : "<datetime>",
"<trigger_type>" : { // type specific data (optional) }
}
}
```
- the structure of `input` results changed to:
```
{
"result" : {
"input" : {
"type" : "<input_type>",
"payload" : { // the payload },
"<input_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `condition` results changed to:
```
{
"result" : {
...
"condition" : {
"type" : "<condition_type>",
"met" : true | false,
"<condition_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `transform` results changed to:
```
{
"result" : {
...
"transform" : {
"type" : "<transform_type>",
"payload" : { // the transformed payload }
"<transform_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `actions` results changed to:
```
{
"result" : {
...
"actions" : [
{
"id" : "<action_id>"
"type" : "<action_type>",
"status" : "success" | "failure" | "simulated" | "throttled",
"reason" : "holds the reasoning if status is either success or throttled",
"transform" : { // action level transform result (if applicable)}
"<action_type>" : { // custom result fields per type (optional) }
},
...
]
}
}
```
Original commit: elastic/x-pack-elasticsearch@98466d4b83
Today, an exception is thrown when calls to the encrypt and decrypt methods are
made without a key being present. For now, we will not require the system key and
this behavior is undesirable.
This commit changes the behavior to just return the provided characters or bytes
when no key is present. Additionally, a method has been added for callers to see
if encryption is supported. Finally, the listener interface has been made public and
expanded to provide the old keys when the keys are changed. This allows
consumers to decrypt with the old key and re-encrypt with the new key.
Original commit: elastic/x-pack-elasticsearch@de3d5b6180
Today, we always throw an AuthorizationException for an unauthorized user. This is
problematic when anonymous access is enabled and the HTTP client being used
does not support preemptive basic authentication as only the anonymous user
will be used by such a client.
This change adds a setting to allow an AuthenticationException to be thrown for
anonymous users. This will clients such as browsers to work with anonymous
access and authenticated access.
Closeselastic/elasticsearch#853
Original commit: elastic/x-pack-elasticsearch@d338b468c7
The esusers tool reads the list of roles to provide validation feedback to the user,
however since we have added custom roles the tool doesn't know about these roles
as they come from outside of Shield. When a custom role was found, a warning
was printed that can be confusing to users.
Now when validating roles, we only read the names from the roles.yml file.
Closeselastic/elasticsearch#835
Original commit: elastic/x-pack-elasticsearch@89d0e3efce
Adds a check to all of the esusers commands to ensure the correct number
of arguments are found. If extra arguments are found, they are printed out
with an error message and the tool exits.
Closeselastic/elasticsearch#817
Original commit: elastic/x-pack-elasticsearch@cd3e786267
Reduces the amount of logging on both HTTP and Transport protocols for
a channel being closed while in the middle of a handshake. This often
occurs when the client does not trust the server certificate and aborts the
handshake. Also, reduces logging on the Transport protocol for a
plain text message received on a channel that is using TLS.
Closeselastic/elasticsearch#771
Original commit: elastic/x-pack-elasticsearch@321c384ddd
This commit removes the commons codec dependency and simplifies the
hasher code by only supporting bcrypt encryption in the users file. All tests
now also exercise the esusers realm with bcrypt instead of plain text passwords.
Closeselastic/elasticsearch#806
Original commit: elastic/x-pack-elasticsearch@3119267851
The queued watches metric gives insight into the watches that are queued for execution.
Per watch that is queued, executing information is shared, like the `watch_id`,
when the watch was triggered and when execution started.
Original commit: elastic/x-pack-elasticsearch@deb5ddfde2
Before WatchRecord was used to keep track of an execution of a Watch and used to store actual end results to it before sealing it. (for example build dashboard on the history indices)
The keeping track of an execution has been moved from WatchRecord to TriggeredWatch. If a watch triggers a TriggeredWatch is stored. The TriggeredWatch has its own index and only the watch_id (is part of id), trigger event and state is stored. If the execution of a Watch has finished (regardless if it was successfully) a WatchRecord is stored in a history index and the TriggeredWatch is deleted.
When a watch is getting executedtThe triggered watch is used the create the watch context.
Also the WatchRecord.State has been removed to its own enum class named ExecutionState. The CHECKING value has been removed, because it wasn't really used. The CHECKING state was set when the execution began, but it was never persisted and because of this state has also been removed from triggered watch.
By separating the result of a watch execution we are more flexible to in the future change the document format of WatchRecord. The history indices will be used by users to build analytics on top of watcher. Also the history indices become truely append only indices.
When update the watch status, only change the status part with the update api
Also set the version when we delete the watch on the in memory instance enforce more ensureStarted() in the components
Removed all watch record and result parsing code (actions, conditions, inputs and transforms)
Original commit: elastic/x-pack-elasticsearch@8f5ffdac13
This commit makes TLSv1.2 the default protocol for better security. The
old value of TLS would only pick a TLSv1.0 supporting context and cause
client connections to be negotiated using that protocol when TLSv1.2 is
supported and considered an improved protocol.
Closeselastic/elasticsearch#867
Original commit: elastic/x-pack-elasticsearch@1f062f6dde
This commit removes the DHE cipher from our list of enabled ciphers
due to the recently published Logjam attack. The default configuration
is not susceptible to the Logjam attack, but since we support Java 7
the maximum prime size (768 bit) is considered too weak. Java 8 supports
1024 bit primes, but these are also not ideal and this cipher should not be
used with a prime smaller than 2048 bits.
Closeselastic/elasticsearch#862
Original commit: elastic/x-pack-elasticsearch@9785bf47cf
- Consolidated setting validation under `WatcherSettingsValidation`
- `WatcherSettingsException` is now only used for settings errors
We need this consolidation as Guice doesn't deal well with exceptions in constructors. So instead, `WatcherSettingsValidation` can be injected and used as a registry for settings errors and then, since it's a service, if there any registered errors, it'll throw `WatcherSettingsException` when it's started.
Fixeselastic/elasticsearch#539
Original commit: elastic/x-pack-elasticsearch@2c1895d18c
The HttpClient class tried to use the truststore as a keystore for watcher with Shield fallback.
This is trappy because in Shield, the default is to only have a keystore that also acts as the
truststore, so the fallback would probably fail in most cases. Additionally, support was missing
for a keystore that had a separate key password.
The new behavior allows for specifying both a keystore and a truststore. If only a keystore is
specified, it will also be used as the truststore. If neither is defined, the system truststore will
be used. All settings fallback to shield settings.
Closeselastic/elasticsearch#480
Original commit: elastic/x-pack-elasticsearch@ad02814806
- now it's possible to ack specific actions via the `Ack Watch API`
- Added tests for acking specific actions
- Changed the watch status structure such that the action ack state can be referred to by `status.actions.<action_id>.ack` (instead of `status.actions.<action_id>.ack_status`... removed the extra redundant "_status")
- As part of this work, also added validation for watch/action ids, such that we disallow having whitespaces in them.
- Updated the docs around acking & throttling of watch actions
Closeselastic/elasticsearch#531Closeselastic/elasticsearch#537
Original commit: elastic/x-pack-elasticsearch@813e601bf5
jaymode
Konrad Beiske
uboness
Martijn van Groningen
aleph-zero
Brian Murphy
Areek Zillur