Commit Graph

1456 Commits

Author SHA1 Message Date
uboness a9448ab2a4 Moved condition errors to their result
Until now if a condition failed to execute (for whatever reason), an exception would be thrown and the watch would just abort. The problem with that approach is while the error message would have been logged in the watch record message, the result of the condition would have been lost.

This commit moves the condition execution error to the condition result (just like we have it today with actions and inputs).

- A `status` field was added to the condition result (can either be `success` or `failure`)
- A `reason` field is added to the condition result in case its status is `failure`
- If the condition result status is `failure`, the watch execution is aborted

Updated the rest APIs to verify the status & type of both the `input` and `condition` results on execution.

Original commit: elastic/x-pack-elasticsearch@dddca03ff5
2015-06-13 04:22:31 +02:00
jaymode f4ed6282fd only bind the index audit trail if auditing is enabled
If auditing is not enabled, we should not bind the index audit trail as a lifecycle component
since we do not bind all of its dependent classes when auditing is disabled.

Original commit: elastic/x-pack-elasticsearch@39f736ca6a
2015-06-12 17:08:31 -04:00
jaymode 95ad77a778 allow periods in esusers usernames and role names
This change allows periods in the usernames and role names in the esusers realm.

Closes elastic/elasticsearch#905

Original commit: elastic/x-pack-elasticsearch@64b4f02ee5
2015-06-12 16:39:29 -04:00
uboness 0ac5dc8271 [lib] - upgraded java mail to 1.5.3
Original commit: elastic/x-pack-elasticsearch@15142f2c79
2015-06-12 21:38:02 +02:00
jaymode 9a97b046d5 control index audit events through an include/excludes mechanism
This changes how the user defines which events should be indexed. Previously, there were
several boolean settings being used. This condenses to an include and exclude setting.

Additionally, the IndexAuditTrail needed to become a lifecycle component since parsing the
enum could throw exceptions. Given this, the IndexBulkProcessor was condensed into the
IndexAuditTrail since it did not make sense to have two lifecycle components for an index
audit trail.

Closes elastic/elasticsearch#900

Original commit: elastic/x-pack-elasticsearch@4b4d824f5e
2015-06-12 14:17:38 -04:00
jaymode c82816dd49 ensure we call authentication failed even if an exception is thrown
Today, a realm could throw an exception, which will not be caught and cause the audit
trail to be skipped. This commit catches exceptions thrown by realms and ensures that
the audit trail is called and then re-throws the exceptions.

Closes elastic/elasticsearch#901

Original commit: elastic/x-pack-elasticsearch@d260c00b33
2015-06-12 12:20:09 -04:00
jaymode 7c67b49aa5 system actions are only logged at the correct audit level
Today, some system actions could be logged by default when the actions performed are
not internal. Additionally for internal actions, we never checked if the user was the system
user.

This adds a check to ensure the user is the System user and that the actions that are being
suppressed are known system actions.

Closes elastic/elasticsearch#902

Original commit: elastic/x-pack-elasticsearch@b107994692
2015-06-12 08:39:43 -04:00
uboness 288ce368d5 Moved input errors to the input result
Until now, if the input failed, an exception would be thrown and it would be captured globally on the watch execution and in the watch recod message. The problem with this approach is that the information about the input is lost. In this commit, the failure is returned as part of the input result.

- A new `status` field was added to the input result. Can either have `success` or `failure` values. When set to `failure` a `reason` field will be set with the error message.
- The `ExecutionService` changed to enable this functionality. Mainly, instead of relying on exception, during the execution the input result is checked for its status and the execution is aborted on failure. Also, the two places where the watch execution is handled were consolidated to a single method `execute(WatchExecutionContext)`.
- Also, the watch execution context id (which will end up being the `watch_record` id) was added the the context model (accessible via scripts and templates). This is done mainly for debugging purposes.

Original commit: elastic/x-pack-elasticsearch@e2567deada
2015-06-12 14:08:07 +02:00
jaymode 5a6dcfa0ac fix compilation due to change in snapshot state enum
Original commit: elastic/x-pack-elasticsearch@771333594d
2015-06-12 06:34:54 -04:00
Konrad Beiske c5a74df643 Refactored api for plugins into it's own module
Original commit: elastic/x-pack-elasticsearch@c3505f6c65
2015-06-11 23:50:55 +02:00
uboness b54b52bb2f Added version 1.0.0-Beta2
Original commit: elastic/x-pack-elasticsearch@ba6dfa6d96
2015-06-10 16:40:15 +02:00
uboness c5fd06d624 [test] added a test to make sure a failed action is not throttled
Closes elastic/elasticsearch#253

Original commit: elastic/x-pack-elasticsearch@89f9075731
2015-06-09 15:37:16 +02:00
jaymode ae3d56f55e [Test] use the new resource watcher settings
Original commit: elastic/x-pack-elasticsearch@09c9951aaf
2015-06-09 08:32:54 -04:00
uboness babe40137f [cleanup] Reorganized the integration tests
Distributed them to the proper packages, such that they'll be easy to find

Original commit: elastic/x-pack-elasticsearch@e6e6c15870
2015-06-09 14:19:47 +02:00
Martijn van Groningen 478d5ecc48 Stop the TriggeredWatchStore and HistoryStore if there are no current executions.
There may be current executions still going on during stopping.
We should try to wait for the current executions to have completed.
Otherwise we can run into a situation where we didn't delete the watch from the .triggered_watches index,
but did insert into the history index. Upon start this can lead to DocumentAlreadyExistsException,
because we already stored the history record during shutdown...

Introduced `CurrentExecutions` to handle this synchronization.

(we always first store the watch record and then remove the triggered watch)

Original commit: elastic/x-pack-elasticsearch@89c4a8d8ad
2015-06-08 20:38:37 +02:00
uboness e684194c23 Fix 1. compatibility
Original commit: elastic/x-pack-elasticsearch@e409f99754
2015-06-08 12:55:14 +02:00
uboness c491e4db16 Added support for multi doc indexing to index action
This commit adds support for indexing multiple documents with the `index` action. This is done by introducing a special `_doc` field. During action execution, the `_doc` field will be looked up in the payload. If found, the value of the field will be considered as the document that needs to be indexed. If the value is an array of objects, each object in that array will be treated as a separate document and all the documents in the array will be bulk indexed.

This commit also changes the result of the action to hold `XContentSource` rather than a payload (to avoid Map creation explosions). Th `XContentSource` was also extended to support lists.

Original commit: elastic/x-pack-elasticsearch@86f454b029
2015-06-07 17:17:38 +02:00
jaymode 018919fb43 fix JDK7 compilation
Original commit: elastic/x-pack-elasticsearch@56e5f69803
2015-06-06 20:26:15 -04:00
uboness ccdaf2f116 compare condition result
The `compare` condition result will now hold all resolved values, keyed by their associated parameter place holder strings.

Original commit: elastic/x-pack-elasticsearch@f930c77d54
2015-06-06 23:12:09 +02:00
jaymode 26f52be95a [Test] add response to the fail statement for debugging
Original commit: elastic/x-pack-elasticsearch@c6628a5cda
2015-06-06 15:48:14 -04:00
jaymode 1c54d49a71 remove use of shaded core packages
Elasticsearch core no longer uses shaded dependencies by default, so this change makes
the plugin use the unshaded dependencies.

Original commit: elastic/x-pack-elasticsearch@495f636266
2015-06-06 15:37:24 -04:00
jaymode a79557c0bc remove use of shaded core packages
Elasticsearch core no longer uses shaded dependencies by default, so this change makes
the plugin use the unshaded dependencies.

Original commit: elastic/x-pack-elasticsearch@6f8b8b3feb
2015-06-06 15:29:44 -04:00
uboness a9658cca77 Enable both timewarp on & off for ActionThrottleTests
Original commit: elastic/x-pack-elasticsearch@caa2da607f
2015-06-06 13:04:50 +02:00
uboness 096bd5b039 Fix throttling test failure
Original commit: elastic/x-pack-elasticsearch@f3799bddeb
2015-06-06 04:03:51 +02:00
uboness 034425185a Fixed failing test
Original commit: elastic/x-pack-elasticsearch@1291299318
2015-06-06 03:33:46 +02:00
aleph-zero a1b4a47684 Store audit events in an index
This commit provides the ability to store audit events into an
Elasticsearch index.

Closes elastic/elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@6e136d8dbb
2015-06-05 18:04:47 -07:00
Brian Murphy 8bf45f0340 [TEST] Watch action acking and throttling tests
This change adds tests to ack a subset of a watch's actions, use a different throttle period per action in a watch, also adds tests to make sure that both the watch level and global level throttle_period are applied correctly.
Also updates the REST tests to make sure that throttle periods can be set at a watch and action level and are returned from the GET API.

Original commit: elastic/x-pack-elasticsearch@4b006c7830
2015-06-06 02:35:42 +02:00
jaymode b713d16803 ensure protocols and ciphers are set on SSLSockets
Today, we simply return a SSLSocketFactory from a SSLContext, which provides
the default SSL configuration for sockets. This means that SSLv3 could still be
enabled on these sockets when running in an older JVM. This also means that
the ciphers and protocol settings are not honored for users of this socket factory,
which is currently the LDAP code.

This change returns a custom socket factory that delegates to the default socket
factory and sets the ciphers and protocols on the socket before returning the
socket.

Original commit: elastic/x-pack-elasticsearch@c4cfedfd51
2015-06-05 16:20:06 -04:00
jaymode 6f079dd2f2 PKI: allow username extraction to work for certs with CN only
Our current CA configuration creates certificates with only a CN and this caused
the regular expression in the PKI realm to fail. The default regular expression in
the PKI realm has been changed to allow for only a CN, while still maintaining the
ability to extract only the CN when other fields are present in the DN.

Additionally, the CA configuration has been updated so that is will copy any of the
originally specified fields (besides CN and EMAILADDRESS) over to the signed
certificate.

Original commit: elastic/x-pack-elasticsearch@ff27f69781
2015-06-05 16:00:06 -04:00
jaymode 7c62e4c82c disable core caches to ensure users are always authorized
The IndicesTermsFilter Cache in core can leak data by not authorizing users prior to
retrieving data from the cache. We work around this by ensuring that the cache has
a maximum size of 0, effectively disabling it.

A test is also added to ensure that data is not leaked by this cache or the cache used by
the ScriptService in core.

Closes elastic/elasticsearch#854

Original commit: elastic/x-pack-elasticsearch@8a48bdad98
2015-06-05 11:28:13 -04:00
jaymode 0f56bd37d8 PKI: add the ability to set want instead of need client authentication
Today, we only have the option to require client authentication or not require it. A third case
exists where the server can want client authentication, but if the client does not have credentials
or wish to send credentials the connection will still be allowed.

This is beneficial for the PKI realm because it allows some clients to authenticate with PKI and
others to use another authentication token such as username and password authentication.

Closes elastic/elasticsearch#858

Original commit: elastic/x-pack-elasticsearch@e7cbc20bca
2015-06-05 09:39:38 -04:00
jaymode 8949e5fcaf fix compilation issue with parsing TimeValue
Original commit: elastic/x-pack-elasticsearch@a7390861be
2015-06-05 09:20:33 -04:00
uboness d899c4b522 Execute API to support default trigger
Until today it was required to provide a trigger event as part of the execute API. There are two issues with it:

1. It's not user friendly (you'd expect that the execute API would just work with just pointing to the watch)
2. The API could expose inconsistencies where on one hand it points to a watch (with a well defined trigger) on the other it enabled the user to provide completely different trigger (of a different type)

This change enable supporting default triggers by enabling the trigger engine to create a simulated trigger event. This enables the execute API to look up the trigger type of the pointed watch, and ask the trigger service to simulate an event for it. It is still possible to override the trigger event data by providing it via the `trigger_data` parameter in the API.

This simplifies the execute API and prepares it for future trigger types as well.

- This commit add missing integration tests for the execute API
- Also, removed unused `setIgnoreThrottling` from the execute request/builder.

Original commit: elastic/x-pack-elasticsearch@b494ae62e6
2015-06-05 12:41:28 +02:00
Areek Zillur c48770c525 fix DateMathParser compilation error
Original commit: elastic/x-pack-elasticsearch@e3cd2e0e45
2015-06-03 14:43:11 -04:00
Brian Murphy 2cff8c4c32 [TEST] Increase logging for long running delete test.
This test is failing on jenkins but does not reproduce.

Original commit: elastic/x-pack-elasticsearch@2f873261c0
2015-06-03 08:43:27 -04:00
Brian Murphy 14ffe290f7 Add support for http input endpoints that do not return JSON formatted bodys.
This change allows the httpinput to receive non json formatted data from a http endpoint
(such as the elasticsearch _cat apis). If non json is read it will be stored in the `payload._value` in the same
way that the `ScriptTransform` handles non map/json data returned by transforming scripts.
Added response_content_type to http input so that the expected content type in the response can be configured. This accepts `yaml`, `json` and `text` but will be overridden by the http headers.

Original commit: elastic/x-pack-elasticsearch@753d37f14e
2015-06-02 13:51:18 -04:00
uboness 613ce8762c De-normalize watch record format
Now that the watch record is write once and never read/parsed. We can de-normalize its format (the  structure of the `watch_record` document) such it'd be optimal for searching and aggregating.

- `execution_result` renamed to `result` (for minimal verbosity)

- the structure of `trigger_event` changed to:
 ```
 {
 	"trigger_event" : {
 		"type" : "<trigger_type>",
 		"triggered_time" : "<datetime>",
 		"<trigger_type>" : { // type specific data (optional) }
 	}
 }
 ```

- the structure of `input` results changed to:
 ```
 {
 	"result" : {
 		"input" : {
 			"type" : "<input_type>",
 			"payload" : { // the payload },
 			"<input_type>" : { // custom result fields per type (optional) }
 		},
 		...
 	}
 }
 ```

- the structure of `condition` results changed to:
 ```
 {
 	"result" : {
 	    ...
 		"condition" : {
 			"type" : "<condition_type>",
 			"met" : true | false,
 			"<condition_type>" : { // custom result fields per type (optional) }
 		},
 		...
 	}
 }
 ```

- the structure of `transform` results changed to:
 ```
 {
 	"result" : {
 	    ...
 		"transform" : {
 			"type" : "<transform_type>",
 			"payload" : { // the transformed payload }
 			"<transform_type>" : { // custom result fields per type (optional) }
 		},
 		...
 	}
 }
 ```

- the structure of `actions` results changed to:
 ```
 {
 	"result" : {
 	    ...
 		"actions" : [
 			{
 				"id" : "<action_id>"
 				"type" : "<action_type>",
 				"status" : "success" | "failure" | "simulated" | "throttled",
 				"reason" : "holds the reasoning if status is either success or throttled",
 				"transform" : { // action level transform result (if applicable)}
 				"<action_type>" : { // custom result fields per type (optional) }
 			},
 			...
 		]
 	}
 }
 ```

Original commit: elastic/x-pack-elasticsearch@98466d4b83
2015-06-02 03:57:21 +02:00
jaymode a12eba49fa make encryption and decryption tolerant to missing key
Today, an exception is thrown when calls to the encrypt and decrypt methods are
made without a key being present. For now, we will not require the system key and
this behavior is undesirable.

This commit changes the behavior to just return the provided characters or bytes
when no key is present. Additionally, a method has been added for callers to see
if encryption is supported. Finally, the listener interface has been made public and
expanded to provide the old keys when the keys are changed. This allows
consumers to decrypt with the old key and re-encrypt with the new key.

Original commit: elastic/x-pack-elasticsearch@de3d5b6180
2015-06-01 13:47:09 -04:00
jaymode 5309353745 allow authentication exceptions for unauthorized anonymous users
Today, we always throw an AuthorizationException for an unauthorized user. This is
problematic when anonymous access is enabled and the HTTP client being used
does not support preemptive basic authentication as only the anonymous user
will be used by such a client.

This change adds a setting to allow an AuthenticationException to be thrown for
anonymous users. This will clients such as browsers to work with anonymous
access and authenticated access.

Closes elastic/elasticsearch#853

Original commit: elastic/x-pack-elasticsearch@d338b468c7
2015-06-01 09:40:55 -04:00
jaymode d393cc2740 do not attempt to resolve permissions in the esusers tool
The esusers tool reads the list of roles to provide validation feedback to the user,
however since we have added custom roles the tool doesn't know about these roles
as they come from outside of Shield. When a custom role was found, a warning
was printed that can be confusing to users.

Now when validating roles, we only read the names from the roles.yml file.

Closes elastic/elasticsearch#835

Original commit: elastic/x-pack-elasticsearch@89d0e3efce
2015-06-01 09:07:10 -04:00
jaymode 03520e0aa7 [CLI] check the number of arguments for esusers commands
Adds a check to all of the esusers commands to ensure the correct number
of arguments are found. If extra arguments are found, they are printed out
with an error message and the tool exits.

Closes elastic/elasticsearch#817

Original commit: elastic/x-pack-elasticsearch@cd3e786267
2015-06-01 08:21:31 -04:00
jaymode b8f75a2bae Netty: reduce logging for a close request during handshake
Reduces the amount of logging on both HTTP and Transport protocols for
a channel being closed while in the middle of a handshake. This often
occurs when the client does not trust the server certificate and aborts the
handshake. Also, reduces logging on the Transport protocol for a
plain text message received on a channel that is using TLS.

Closes elastic/elasticsearch#771

Original commit: elastic/x-pack-elasticsearch@321c384ddd
2015-06-01 08:04:55 -04:00
jaymode 6e660dbd7d remove commons codec dependency
This commit removes the commons codec dependency and simplifies the
hasher code by only supporting bcrypt encryption in the users file. All tests
now also exercise the esusers realm with bcrypt instead of plain text passwords.

Closes elastic/elasticsearch#806

Original commit: elastic/x-pack-elasticsearch@3119267851
2015-06-01 07:52:38 -04:00
Martijn van Groningen 47247dc46a Stats api: fix serialization issue
Renamed pending watches left overs to queued watches

Original commit: elastic/x-pack-elasticsearch@c2bcdf547c
2015-05-31 13:14:32 +02:00
Martijn van Groningen 0890001470 stats: Add queued watches metric
The queued watches metric gives insight into the watches that are queued for execution.
Per watch that is queued, executing information is shared, like the `watch_id`,
when the watch was triggered and when execution started.

Original commit: elastic/x-pack-elasticsearch@deb5ddfde2
2015-05-31 11:13:13 +02:00
Martijn van Groningen 4335669635 Refactoring: Introduced TriggeredWatch concept.
Before WatchRecord was used to keep track of an execution of a Watch and used to store actual end results to it before sealing it. (for example build dashboard on the history indices)

The keeping track of an execution has been moved from WatchRecord to TriggeredWatch. If a watch triggers a TriggeredWatch is stored. The TriggeredWatch has its own index and only the watch_id (is part of id), trigger event and state is stored. If the execution of a Watch has finished (regardless if it was successfully) a WatchRecord is stored in a history index and the TriggeredWatch is deleted.

When a watch is getting executedtThe triggered watch is used the create the watch context.

Also the WatchRecord.State has been removed to its own enum class named ExecutionState. The CHECKING value has been removed, because it wasn't really used. The CHECKING state was set when the execution began, but it was never persisted and because of this state has also been removed from triggered watch.

By separating the result of a watch execution we are more flexible to in the future change the document format of WatchRecord. The history indices will be used by users to build analytics on top of watcher. Also the history indices become truely append only indices.

When update the watch status, only change the status part with the update api
Also set the version when we delete the watch on the in memory instance enforce more ensureStarted() in the components

Removed all watch record and result parsing code (actions, conditions, inputs and transforms)

Original commit: elastic/x-pack-elasticsearch@8f5ffdac13
2015-05-30 17:59:24 +02:00
Areek Zillur 900ea3cd20 fix XContextSerialization test failure
Original commit: elastic/x-pack-elasticsearch@f2e1b19214
2015-05-29 16:29:13 -04:00
jaymode ba1001a3a4 update BCrypt implementation to version 0.4
This updates the BCrypt implementation that we use to version 0.4, which
corrects an integer overflow bug when a large number of rounds are used.

Closes elastic/elasticsearch#865

Original commit: elastic/x-pack-elasticsearch@2f9a07e7c8
2015-05-29 13:52:39 -04:00
jaymode 2e4f3e8d23 make TLSv1.2 the default protocol
This commit makes TLSv1.2 the default protocol for better security. The
old value of TLS would only pick a TLSv1.0 supporting context and cause
client connections to be negotiated using that protocol when TLSv1.2 is
supported and considered an improved protocol.

Closes elastic/elasticsearch#867

Original commit: elastic/x-pack-elasticsearch@1f062f6dde
2015-05-29 13:46:48 -04:00
jaymode 44017711e2 remove DHE cipher from default list
This commit removes the DHE cipher from our list of enabled ciphers
due to the recently published Logjam attack. The default configuration
is not susceptible to the Logjam attack, but since we support Java 7
the maximum prime size (768 bit) is considered too weak. Java 8 supports
1024 bit primes, but these are also not ideal and this cipher should not be
used with a prime smaller than 2048 bits.

Closes elastic/elasticsearch#862

Original commit: elastic/x-pack-elasticsearch@9785bf47cf
2015-05-29 13:33:54 -04:00