Commit Graph

701 Commits

Author SHA1 Message Date
uboness 8e13a03069 fixing failing test on windows... now for realz
Original commit: elastic/x-pack-elasticsearch@159144f18c
2015-05-22 00:41:32 +02:00
uboness 47e50008a0 fixing failing test on windows
Original commit: elastic/x-pack-elasticsearch@5af44b88d2
2015-05-21 22:00:41 +02:00
Martijn van Groningen 1bdd84bcb2 test: Add a simple runner that allows to run Watcher from the IDE and work out of the box with Sense.
Original commit: elastic/x-pack-elasticsearch@8c4100258e
2015-05-21 14:44:13 +02:00
Martijn van Groningen bbcfecbe90 test: added 'cluster:monitor/nodes/info' role for the test user
Original commit: elastic/x-pack-elasticsearch@78144d3313
2015-05-21 13:35:19 +02:00
Martijn van Groningen 39f83974ef test: added missing roles for the test user
Original commit: elastic/x-pack-elasticsearch@36f6d36817
2015-05-21 12:41:30 +02:00
Martijn van Groningen 2861f8ce21 test: Add more Shield related tests for testing the Watcher roles.
Original commit: elastic/x-pack-elasticsearch@482d8fe65c
2015-05-21 12:05:44 +02:00
Martijn van Groningen 284a60e16d stats: fix npe during serialization
Original commit: elastic/x-pack-elasticsearch@3233836d0b
2015-05-21 11:38:24 +02:00
Martijn van Groningen 60c7e92ced stats: fix serialization error
Original commit: elastic/x-pack-elasticsearch@5210ca9ee7
2015-05-21 11:29:32 +02:00
Martijn van Groningen 283445b9d8 stats: fix npe in rest layer
Original commit: elastic/x-pack-elasticsearch@43b3fcf3e7
2015-05-21 11:12:52 +02:00
Martijn van Groningen c3dd74df7f Expose the current executing watches as part of the Watcher stats API.
The following additional information will be shown per watch that is executing:  `watch_id`, `watch_execution_id`, `triggered_time`, `execution_time`, `execution_phase` (whether it is execution an input, condition or an action) and `stack_trace` (useful for us when a customer reports an issue :) ).

The stats api will by default include the executing watches in the response. In order to control this, a `metric` option has been added, which can be specified as query string argument or as last path element in the stats api url. By default the watcher stats API will only return the basic statistics that are already there.

The `metric` option has the following values:
* `current_watches` - Include the current executing watches in the response.
* `_all` - Include all metrics in the stats response. Not very useful now, but when we expose more metrics in this api it will be more useful.

Original commit: elastic/x-pack-elasticsearch@093bef9bb3
2015-05-21 10:51:51 +02:00
Brian Murphy 81d19d3468 [REST-TEST] Remove check for snapshot.
This change removes the check for snapshot in the info test.

Original commit: elastic/x-pack-elasticsearch@5eb0618b56
2015-05-20 09:58:05 -04:00
uboness 7d12d314ad moving version to 2.0.0-SNAPSHOT
Original commit: elastic/x-pack-elasticsearch@58e7b38de5
2015-05-20 15:29:52 +02:00
uboness 0a3c85803c moving version to 1.0.0-Beta1
Original commit: elastic/x-pack-elasticsearch@d88dd97379
2015-05-20 14:23:05 +02:00
uboness 68d0420dd2 Added maven deployment profiles
Original commit: elastic/x-pack-elasticsearch@37a867aa26
2015-05-20 13:36:36 +02:00
uboness 8218170711 Added `execution_result.condition.met` field
Until today we could not search on the `met` field in the condition result. The reason for that is that this field was index as part of the condition result type only, and we disable the indexing for all condition results (to avoid mapping conflicts).

 This commit pulls the `met` condition one level higher and enables its mapping. For now (beta1) we can live with the duplication of the condition result source (were the `met` is not placed in both the condition result type and on the condition result itself). Later we should remove the duplication though.

 An example of a  "compare" condition result now looks like:

 ```
 "condition": {
    "met": true,
    "compare": {
       "met": true,
       "resolved_value": 1
    }
 }
 ```

Original commit: elastic/x-pack-elasticsearch@74a3372c25
2015-05-20 09:46:42 +02:00
Martijn van Groningen 2ede3c29d8 test: removed unused method
Original commit: elastic/x-pack-elasticsearch@465bf69daa
2015-05-19 16:58:08 +02:00
uboness 9aef7bb52b Added `compare` condition
A simple `condition` that compares a path into the model in the execution context to a value. The comparison is based on the following possible operators: `eq`, `not_eq`, `lt`, `lte`, `gt`, `gte`.

The following example shows a `compare` condition that checks if the total hits in the payload is greater or equal to 5.

```
{
	"compare" : {
		"ctx.payload.hits.total" : { "gte" :  "5" }
	}
}
```

Original commit: elastic/x-pack-elasticsearch@6d4f2bbf10
2015-05-19 04:17:12 +02:00
uboness 9e5fa64e03 Fixed watch_history template for Index Action
Fixed the mappings of the index action fields such that the `index` and `type` of the index response won't be analyzed.

Original commit: elastic/x-pack-elasticsearch@5db8bf6a33
2015-05-19 03:11:56 +02:00
Martijn van Groningen 4b3e1592ca Test: apply `shield.user` property only once,
rely on http header to be send for Shield authentication
and finally omit the manage_watcher role from the test user.

Original commit: elastic/x-pack-elasticsearch@57a6366119
2015-05-18 18:47:06 +02:00
Martijn van Groningen 00c5c5c8b6 Test: Added a test that verifies that we fail with 403 when the user doesn't have the `manage_watcher` role.
Original commit: elastic/x-pack-elasticsearch@4776bbdd11
2015-05-18 17:16:00 +02:00
Martijn van Groningen 511ac9f6a9 Test: Use internalTestCluster() instead of internalCluster()
Original commit: elastic/x-pack-elasticsearch@a2b72000c8
2015-05-18 13:26:46 +02:00
Martijn van Groningen 6404b2c7e5 Test: clear unicast temporal response list before we stop the elected master node.
Original commit: elastic/x-pack-elasticsearch@0b2675f9ff
2015-05-18 13:06:42 +02:00
Martijn van Groningen 1db5d1e37d Test: fix build failure where randomly a restricted header was used.
Original commit: elastic/x-pack-elasticsearch@c3892181ab
2015-05-18 12:41:00 +02:00
uboness b11f457fb7 Updated the watch_history template
- Disabled the search request body/template fields as they're too volatile to have mappings (applied for both for `search` input and transform)
- Disabled watch level transform result payload

Original commit: elastic/x-pack-elasticsearch@f69b237234
2015-05-16 02:13:17 +02:00
Martijn van Groningen a52a833264 test: change test logging
Original commit: elastic/x-pack-elasticsearch@2fe4619b4f
2015-05-15 12:50:36 +02:00
uboness 912bd1299b Introduced version name to the info API
Holds the name of the version as it is defined in the `pom.xml` (different than the version number that is automatically generated by `WatcherVersion`)

Original commit: elastic/x-pack-elasticsearch@b6cf221f63
2015-05-15 04:54:33 +02:00
uboness ba5f25f73c Updated `WatcherVersion` with min es compatibility to 1.5
Original commit: elastic/x-pack-elasticsearch@a48048aae4
2015-05-15 03:07:36 +02:00
Brian Murphy 7c1497ae3b [TEST] Ensure the license is enabled when starting watcher
This change adds a check to make sure that the license is enabled when starting watcher in the tests.
This avoids a race-condition where a test might run before the license manager had a chance to start.

Original commit: elastic/x-pack-elasticsearch@0b9d0da5d4
2015-05-14 18:22:09 -04:00
Brian Murphy 54926ec336 Fix escaping of mustache strings.
I think the escaping done in XMustacheFactory (and by extension JsonEscapingMustacheFactory in core) is broken.
You cannot just escape any control character by sticking a '\' in front of it. For example a new line character it '\n' but this will be rendered as a line break. Simply prepending a '\' to this just results in a '\' and then a new line !
Added support for different escaping strategies based on the XContentType of a template for XMustacheEngine.
Currently only JSON escaping is supported using jackson.JsonStringEncoder.
Templates will be prepended with __<contentType>__:: when the content type is set. If this is set to JSON we will json escape the content.

Fixes: elastic/elasticsearch#404

Original commit: elastic/x-pack-elasticsearch@1400cba659
2015-05-14 14:35:16 -04:00
jaymode 09621f1267 allow the system truststore to be used if one is not specified
A truststore should not be required as the default system truststore can be used
to validate certificates that have been signed by most commercial CAs.

Additionally, the HttpClient is now a lifecycle component to prevent out of memory
exceptions when starting up with a bad configuration; when an exception is thrown
in the constructor, Guice will continue to try to create the object until the system runs
out of memory.

Closes elastic/elasticsearch#476

Original commit: elastic/x-pack-elasticsearch@2333e47ac1
2015-05-14 09:45:43 -04:00
Brian Murphy 5796a771c0 Chain transform was incorrectly assuming that all transforms had to be objects.
The chain transform :
```
  "transform" : {
    "chain" : [
      {
        "search" : {
          "search_type" : "count",
          "indices" : [ "logstash-*" ],
          "body" : {
            "query" : {
              "match" : { "priority" : "error"}
            }
          }
        }
      },
      {
         "script" :  "return [ error_count : ctx.payload.hits.total ]"

      }
    ]
  },
```
Was failing because the chain transform (wrongly) refused to parse the script string as a transform.
This change will allow the transforms to determine what should parse or not.

Fixes: elastic/elasticsearch#475

Updates after review

Original commit: elastic/x-pack-elasticsearch@78c1a2498d
2015-05-13 22:23:08 -04:00
uboness 926e39d21e Disabled mapping for transform results
Transform results may product different payloads per watch/execution. It means that if the resulted transformed payload is mapped and indexed, there's a high chance for mapping conflicts and thus failures.

For this reason, this commit disables the mapping of all `transform` results (on both the watch and the action levels).

This commit also changes the field name of the transform result from `transform_result` to just `transform` (aligned with other result field names - `input`, `condition` and `actions`)

 Fixes elastic/elasticsearch#472

Original commit: elastic/x-pack-elasticsearch@2c6d4f5182
2015-05-13 20:18:45 +02:00
Martijn van Groningen aba3f8f34c Revert TP SAME change for the service api
Original commit: elastic/x-pack-elasticsearch@f7e52fda96
2015-05-13 14:03:10 +02:00
Martijn van Groningen af1d4ad79c Use ThreadPool.Names.SAME executor in stats and service apis.
Original commit: elastic/x-pack-elasticsearch@03c52668e0
2015-05-13 10:44:15 +02:00
Brian Murphy f14fe49dfa Use a different separator in `Wid`
`#` is a reserved character in the URL spec and must be escaped to be used. This change uses `-` instead.

Fixes: elastic/elasticsearch#467

Original commit: elastic/x-pack-elasticsearch@a75eb89cb8
2015-05-12 17:26:28 -04:00
uboness a59e157ca3 Added default `actions`
Until now you always needed to define the `actions` in a watch. Even if you didn't want any actions, you had to define and empty objects there.

Now, the `actions` is an optional field. When missing, we default to "no actions"

Original commit: elastic/x-pack-elasticsearch@3d1a961232
2015-05-12 20:38:30 +02:00
Brian Murphy 86262d5ca5 [TEST] Fix the MonthlyScheduleTests
Move deduping to `validMonthTimes`
This change dedups the `MonthTimes` in `validMonthTimes` to support this two new methods on `MonthTimes` are added `contains(int,DayTime)` which detects if a given day and DayTime combination is contained in the MonthTime and `intersects(MonthTimes)` which detects if a `MonthTimes` has any day/DayTime combinations contained in this `MonthTimes`.

Original commit: elastic/x-pack-elasticsearch@f68039bc39
2015-05-12 14:10:35 -04:00
Martijn van Groningen 9367cd23f2 test: increasing test logging from trace to debug
Original commit: elastic/x-pack-elasticsearch@8709256210
2015-05-12 14:49:30 +02:00
Bill Hwang 8545ddbf4d Watcher randomization testing
Creation of randomization data file for CI

https://github.com/elastic/infra/issues/188

Original commit: elastic/x-pack-elasticsearch@f201b144a5
2015-05-11 13:33:18 -07:00
Martijn van Groningen 887e97cb81 Merge pull request elastic/elasticsearch#452 from martijnvg/fix_es1x_compatibility
Fix Watcher binary compatibility between ES 1.5 and 1.6

Original commit: elastic/x-pack-elasticsearch@c7a5fc9379
2015-05-11 21:57:17 +02:00
Martijn van Groningen ddfb05dc05 test: properly set node types.
Original commit: elastic/x-pack-elasticsearch@46096c087b
2015-05-11 17:39:28 +02:00
Martijn van Groningen 66c032f6df Fix Watcher binary compatibility between ES 1.5 and 1.6
Original commit: elastic/x-pack-elasticsearch@a367d016aa
2015-05-11 15:32:31 +02:00
uboness 6320f39023 Cleanup for xcontent results + docs
Code:

- renamed 'executed_request` back to `request` in the search input/transform result (makes more sense in the context of the overall response structure
- renamed `sent_request` back to `request` in the http input result (makes more sense in the context of the overall response structure)
- renamed `http_status` to `status` in the http input result (makes more sense in the context of the overall response structure)

Docs:

- fixed indentations
- fixed wrong example in put-watch docs (`actions` was shown as an array)
- changed action ids to be lowercased & underscored

Original commit: elastic/x-pack-elasticsearch@6ded0936d5
2015-05-11 10:54:41 +02:00
uboness 46dfa059fd Changed `watch_record` xcontent structure
- Renamed `watch_execution` to `execution_result`
- Renamed `actions_results` to `actions`
- Renamed `input_result` to `input`
- Renamed `condition_result` to `condition`
- Updated the `watch_history.json` template to reflect the changes, also added concrete mappings for action types (such that field that should not be analized will be mapped as `not_analyzed`
- Fixed a bug in `WatchUtils.createSearchRequestFromPrototype` where the document types were ignored.

Also, changed the `actions` (fka `actions_results`) from an object to an array. the action id is not part of the action objects (indicated by the `id` field). For example:

```
{
  "actions" : [
    {
      "id" : "my_email",
      "email" : {
        ...
      }
    }
  ]
}
```

The reason for this change is to make the path to the action fields predictable deterministic. With the object structure, where the actions were keyed by their `id`, the path to the action fields depended on the action id, which is unpredictable and continuously changing from one action to another. This made it impossible to properly analyze the action data using aggregations (as aggs require full path into the aggregated fields).

With this change, the mappings of `watch_record` changed as well where the `actions` are not defined as nested type, yet it is still configured to include all the fields in the root object. We do this so in the future, when appropriate support will be added to kibana, it'll be able to apply nested aggregations on the actions, enabling correct/safe multi-dimensional aggregations. In the mean time however, while kibana doesn't support nested aggregations, we still need to have all the fields indexed on the root, so at least a single dimensional aggregations can be safely applied.

Also, change the `input` and `condition` objects in the `watch_record` such that their mappings are disabled. The main reason for this is the fact that a lot of the inputs use elements that can be configured in many ways, but the mappings are too strict to accept it. For example, a template can be configured as a `string` or as an `object`.

Original commit: elastic/x-pack-elasticsearch@83464a0c71
2015-05-10 19:17:09 +02:00
Shay Banon 7ff92c8f96 Merge pull request elastic/elasticsearch#445 from kimchy/remove_content_unsafe
Remove rest content unsafe usage

Original commit: elastic/x-pack-elasticsearch@c290326f3f
2015-05-09 16:26:11 +02:00
Shay Banon 02ad78743e Remove rest content unsafe usage
Original commit: elastic/x-pack-elasticsearch@4725a3de22
2015-05-09 16:21:45 +02:00
Brian Murphy 9a5455f573 [TESTS] Fix Inline template tests
This change adds a method to WatcherTestUtils to compare two json structures since order is not guaranteed in template parameters.
The Inline template tests in SearchInputTests and SearchTransformTests now use this instead of string comparision.

Original commit: elastic/x-pack-elasticsearch@c433545128
2015-05-09 10:18:45 -04:00
Martijn van Groningen 8fad9937f7 Changed search template format for search input and search transform to the new Watcher template format.
Also the WatcherUtils now always use the search template support from core and never uses ScriptService:
* A template is re-parsed and extended with the watch context variables.
* A normal request body is converted into a search template and watch context variables are used as template params.

Original commit: elastic/x-pack-elasticsearch@16bacaf094
2015-05-09 13:17:25 +02:00
uboness 8ddb0a65c4 Added a setting to disable watcher
The following node setting will disable watcher (it's enabled by default).

```
watcher.enabled: false
```

Original commit: elastic/x-pack-elasticsearch@1d0541a924
2015-05-09 01:33:00 +02:00
Brian Murphy eef3a1b1e8 Add force parameter to delete to force a delete
This change adds a force paramter to delete to allow the forcing of a delete to happen ignoring locking.
This means that watch executions may fail when they go to update the watch. Watches executing from the scheduler will fail fast if the watch they are supposed to execute has been removed.
Also move the history store updates outside of the watch locks.

Fixes: elastic/elasticsearch#405

Original commit: elastic/x-pack-elasticsearch@57561b6f85
2015-05-08 19:07:10 -04:00