This change adds tests to ack a subset of a watch's actions, use a different throttle period per action in a watch, also adds tests to make sure that both the watch level and global level throttle_period are applied correctly.
Also updates the REST tests to make sure that throttle periods can be set at a watch and action level and are returned from the GET API.
Original commit: elastic/x-pack-elasticsearch@4b006c7830
Today, we simply return a SSLSocketFactory from a SSLContext, which provides
the default SSL configuration for sockets. This means that SSLv3 could still be
enabled on these sockets when running in an older JVM. This also means that
the ciphers and protocol settings are not honored for users of this socket factory,
which is currently the LDAP code.
This change returns a custom socket factory that delegates to the default socket
factory and sets the ciphers and protocols on the socket before returning the
socket.
Original commit: elastic/x-pack-elasticsearch@c4cfedfd51
Our current CA configuration creates certificates with only a CN and this caused
the regular expression in the PKI realm to fail. The default regular expression in
the PKI realm has been changed to allow for only a CN, while still maintaining the
ability to extract only the CN when other fields are present in the DN.
Additionally, the CA configuration has been updated so that is will copy any of the
originally specified fields (besides CN and EMAILADDRESS) over to the signed
certificate.
Original commit: elastic/x-pack-elasticsearch@ff27f69781
The IndicesTermsFilter Cache in core can leak data by not authorizing users prior to
retrieving data from the cache. We work around this by ensuring that the cache has
a maximum size of 0, effectively disabling it.
A test is also added to ensure that data is not leaked by this cache or the cache used by
the ScriptService in core.
Closeselastic/elasticsearch#854
Original commit: elastic/x-pack-elasticsearch@8a48bdad98
Today, we only have the option to require client authentication or not require it. A third case
exists where the server can want client authentication, but if the client does not have credentials
or wish to send credentials the connection will still be allowed.
This is beneficial for the PKI realm because it allows some clients to authenticate with PKI and
others to use another authentication token such as username and password authentication.
Closeselastic/elasticsearch#858
Original commit: elastic/x-pack-elasticsearch@e7cbc20bca
Until today it was required to provide a trigger event as part of the execute API. There are two issues with it:
1. It's not user friendly (you'd expect that the execute API would just work with just pointing to the watch)
2. The API could expose inconsistencies where on one hand it points to a watch (with a well defined trigger) on the other it enabled the user to provide completely different trigger (of a different type)
This change enable supporting default triggers by enabling the trigger engine to create a simulated trigger event. This enables the execute API to look up the trigger type of the pointed watch, and ask the trigger service to simulate an event for it. It is still possible to override the trigger event data by providing it via the `trigger_data` parameter in the API.
This simplifies the execute API and prepares it for future trigger types as well.
- This commit add missing integration tests for the execute API
- Also, removed unused `setIgnoreThrottling` from the execute request/builder.
Original commit: elastic/x-pack-elasticsearch@b494ae62e6
This change allows the httpinput to receive non json formatted data from a http endpoint
(such as the elasticsearch _cat apis). If non json is read it will be stored in the `payload._value` in the same
way that the `ScriptTransform` handles non map/json data returned by transforming scripts.
Added response_content_type to http input so that the expected content type in the response can be configured. This accepts `yaml`, `json` and `text` but will be overridden by the http headers.
Original commit: elastic/x-pack-elasticsearch@753d37f14e
Now that the watch record is write once and never read/parsed. We can de-normalize its format (the structure of the `watch_record` document) such it'd be optimal for searching and aggregating.
- `execution_result` renamed to `result` (for minimal verbosity)
- the structure of `trigger_event` changed to:
```
{
"trigger_event" : {
"type" : "<trigger_type>",
"triggered_time" : "<datetime>",
"<trigger_type>" : { // type specific data (optional) }
}
}
```
- the structure of `input` results changed to:
```
{
"result" : {
"input" : {
"type" : "<input_type>",
"payload" : { // the payload },
"<input_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `condition` results changed to:
```
{
"result" : {
...
"condition" : {
"type" : "<condition_type>",
"met" : true | false,
"<condition_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `transform` results changed to:
```
{
"result" : {
...
"transform" : {
"type" : "<transform_type>",
"payload" : { // the transformed payload }
"<transform_type>" : { // custom result fields per type (optional) }
},
...
}
}
```
- the structure of `actions` results changed to:
```
{
"result" : {
...
"actions" : [
{
"id" : "<action_id>"
"type" : "<action_type>",
"status" : "success" | "failure" | "simulated" | "throttled",
"reason" : "holds the reasoning if status is either success or throttled",
"transform" : { // action level transform result (if applicable)}
"<action_type>" : { // custom result fields per type (optional) }
},
...
]
}
}
```
Original commit: elastic/x-pack-elasticsearch@98466d4b83
Today, an exception is thrown when calls to the encrypt and decrypt methods are
made without a key being present. For now, we will not require the system key and
this behavior is undesirable.
This commit changes the behavior to just return the provided characters or bytes
when no key is present. Additionally, a method has been added for callers to see
if encryption is supported. Finally, the listener interface has been made public and
expanded to provide the old keys when the keys are changed. This allows
consumers to decrypt with the old key and re-encrypt with the new key.
Original commit: elastic/x-pack-elasticsearch@de3d5b6180
Today, we always throw an AuthorizationException for an unauthorized user. This is
problematic when anonymous access is enabled and the HTTP client being used
does not support preemptive basic authentication as only the anonymous user
will be used by such a client.
This change adds a setting to allow an AuthenticationException to be thrown for
anonymous users. This will clients such as browsers to work with anonymous
access and authenticated access.
Closeselastic/elasticsearch#853
Original commit: elastic/x-pack-elasticsearch@d338b468c7
The esusers tool reads the list of roles to provide validation feedback to the user,
however since we have added custom roles the tool doesn't know about these roles
as they come from outside of Shield. When a custom role was found, a warning
was printed that can be confusing to users.
Now when validating roles, we only read the names from the roles.yml file.
Closeselastic/elasticsearch#835
Original commit: elastic/x-pack-elasticsearch@89d0e3efce
Adds a check to all of the esusers commands to ensure the correct number
of arguments are found. If extra arguments are found, they are printed out
with an error message and the tool exits.
Closeselastic/elasticsearch#817
Original commit: elastic/x-pack-elasticsearch@cd3e786267
Reduces the amount of logging on both HTTP and Transport protocols for
a channel being closed while in the middle of a handshake. This often
occurs when the client does not trust the server certificate and aborts the
handshake. Also, reduces logging on the Transport protocol for a
plain text message received on a channel that is using TLS.
Closeselastic/elasticsearch#771
Original commit: elastic/x-pack-elasticsearch@321c384ddd
This commit removes the commons codec dependency and simplifies the
hasher code by only supporting bcrypt encryption in the users file. All tests
now also exercise the esusers realm with bcrypt instead of plain text passwords.
Closeselastic/elasticsearch#806
Original commit: elastic/x-pack-elasticsearch@3119267851
The queued watches metric gives insight into the watches that are queued for execution.
Per watch that is queued, executing information is shared, like the `watch_id`,
when the watch was triggered and when execution started.
Original commit: elastic/x-pack-elasticsearch@deb5ddfde2
Before WatchRecord was used to keep track of an execution of a Watch and used to store actual end results to it before sealing it. (for example build dashboard on the history indices)
The keeping track of an execution has been moved from WatchRecord to TriggeredWatch. If a watch triggers a TriggeredWatch is stored. The TriggeredWatch has its own index and only the watch_id (is part of id), trigger event and state is stored. If the execution of a Watch has finished (regardless if it was successfully) a WatchRecord is stored in a history index and the TriggeredWatch is deleted.
When a watch is getting executedtThe triggered watch is used the create the watch context.
Also the WatchRecord.State has been removed to its own enum class named ExecutionState. The CHECKING value has been removed, because it wasn't really used. The CHECKING state was set when the execution began, but it was never persisted and because of this state has also been removed from triggered watch.
By separating the result of a watch execution we are more flexible to in the future change the document format of WatchRecord. The history indices will be used by users to build analytics on top of watcher. Also the history indices become truely append only indices.
When update the watch status, only change the status part with the update api
Also set the version when we delete the watch on the in memory instance enforce more ensureStarted() in the components
Removed all watch record and result parsing code (actions, conditions, inputs and transforms)
Original commit: elastic/x-pack-elasticsearch@8f5ffdac13
This commit makes TLSv1.2 the default protocol for better security. The
old value of TLS would only pick a TLSv1.0 supporting context and cause
client connections to be negotiated using that protocol when TLSv1.2 is
supported and considered an improved protocol.
Closeselastic/elasticsearch#867
Original commit: elastic/x-pack-elasticsearch@1f062f6dde
This commit removes the DHE cipher from our list of enabled ciphers
due to the recently published Logjam attack. The default configuration
is not susceptible to the Logjam attack, but since we support Java 7
the maximum prime size (768 bit) is considered too weak. Java 8 supports
1024 bit primes, but these are also not ideal and this cipher should not be
used with a prime smaller than 2048 bits.
Closeselastic/elasticsearch#862
Original commit: elastic/x-pack-elasticsearch@9785bf47cf
- Consolidated setting validation under `WatcherSettingsValidation`
- `WatcherSettingsException` is now only used for settings errors
We need this consolidation as Guice doesn't deal well with exceptions in constructors. So instead, `WatcherSettingsValidation` can be injected and used as a registry for settings errors and then, since it's a service, if there any registered errors, it'll throw `WatcherSettingsException` when it's started.
Fixeselastic/elasticsearch#539
Original commit: elastic/x-pack-elasticsearch@2c1895d18c
The HttpClient class tried to use the truststore as a keystore for watcher with Shield fallback.
This is trappy because in Shield, the default is to only have a keystore that also acts as the
truststore, so the fallback would probably fail in most cases. Additionally, support was missing
for a keystore that had a separate key password.
The new behavior allows for specifying both a keystore and a truststore. If only a keystore is
specified, it will also be used as the truststore. If neither is defined, the system truststore will
be used. All settings fallback to shield settings.
Closeselastic/elasticsearch#480
Original commit: elastic/x-pack-elasticsearch@ad02814806
- now it's possible to ack specific actions via the `Ack Watch API`
- Added tests for acking specific actions
- Changed the watch status structure such that the action ack state can be referred to by `status.actions.<action_id>.ack` (instead of `status.actions.<action_id>.ack_status`... removed the extra redundant "_status")
- As part of this work, also added validation for watch/action ids, such that we disallow having whitespaces in them.
- Updated the docs around acking & throttling of watch actions
Closeselastic/elasticsearch#531Closeselastic/elasticsearch#537
Original commit: elastic/x-pack-elasticsearch@813e601bf5
Adds JNA as a test time dependency to work around it being required on
windows in elasticsearch core.
Original commit: elastic/x-pack-elasticsearch@b8fc70a051
This commit is a squashed commit of work done in conjunction with @rmuir to make
shield work properly with elasticsearch 2.0. This includes ensuring all tests
pass when running with the security manager and updating the code to be compatible
with the latest core forbidden apis.
Shield is now a child project of elasticsearch-parent and inherits a lot of common
build, test, and static coverage functionality from that project resulting in a much
smaller and simpler pom.xml for shield. As part of this work, Shield can now read
rest tests from the elasticsearch jar so the rest tests are run automatically when
running the slow tests.
Original commit: elastic/x-pack-elasticsearch@2ebbf4284a
- HttpResponse now holds the response headers
- Added specific support for content type of the response, based on which we create the xcontent payload.
Original commit: elastic/x-pack-elasticsearch@beae27f576
This change add the actual length of time a watch spends executing. This is useful to find watches that take long to execute to pinpoint those watches that may be candidates for throttling.
Add the execution_duration as a number of milliseconds rather than a timevalue so it can be aggregated from the watch_history index.
Original commit: elastic/x-pack-elasticsearch@0036468f55
Until now, acking and throttling functionality was applied at the watch level. This has major drawbacks in different aspects:
- When multiple actions are defined on a watch, acking a watch effectively acks all the actions. This is conceptually wrong. Say you have two actions: `email` and `index`. It's very likely you'd like to ack the email action (to avoid receiving too many emails) but at the same time continue indexing the data in the `index` action. Right now it's not possible.
- Different actions types may require different throttling. An `email` action probably needs a longer throttle period compared to an `index` action. Also for different `webhook` actions, the throttling is ultimately determined by the 3rd party system that is called.
This commit changes how we do throttling & acking. Moving this functionality to the action level. Now, when acking, each action in the watch will be acked separately. During executiong, each action will determine whether it needs to be throttled or not. The throttler is not associated with the action, not with the watch.
The throttle period was enhanced. There is a default throttle period that is configured for watcher as a whole (using the `watcher.execution.default_throttle_period` setting. Next to that, each `watch` can define its own `throttle_period` that can serve as the default throttle period for the actions in the watch. Lastly, each action can have its own throttle period set.
Since the throttler is now an action "thing", the `throttle` package was renamed to `throttler` and moved under the `actions` package. Also, `WatchThrottler` was renamed to `ActionThrottler`.
With this change, the `Watch Execute API` changed as well. Now, when executing a watch, you can define an execution mode per action. The execution mode offers 4 types of execution:
- `execute`: executes the watch normally (actually executing the action and it may be throttled)
- `force_execute`: skips/ignores throttling and executes the watch
- `simulate`: simulates the watch execution yet it may be throttled
- `force_simulate`: skips/ignores throttling and simulates the watch execution
As part of this change, the structure of the watch status changed along with the xconent representing the `watch_record`. A new `ActionStatus` was introduced (as part of the `WatchStatus`) and is always set for every action in the watch. This status holds:
- the current state of the action (`ackable`, `awaits_successful_execution`, `acked`)
- the last execution state (success/failure + reason)
- the last successful execution state
- the last throttle state (timestamp + reason)
Original commit: elastic/x-pack-elasticsearch@32c2985ed8
uboness
aleph-zero
Brian Murphy
jaymode
Areek Zillur
Chris Cowan
Martijn van Groningen
Robert Muir