Commit Graph

514 Commits

Author SHA1 Message Date
Martijn van Groningen 70cc41d710 Changed internal indices naming pattern
The `.watch_history` indices names should be aligned with logstash and marvel indices patterns so changing them to `.watch_history-YY.MM.dd`

Closes elastic/elasticsearch#387

Original commit: elastic/x-pack-elasticsearch@20baa1018f
2015-05-06 11:16:14 +02:00
uboness 5687c91bf6 Update README.asciidoc
Original commit: elastic/x-pack-elasticsearch@f61bd59d86
2015-05-05 23:38:55 +02:00
uboness 690641d0c9 Cleaned up the `Script` and `Template` constructs
Change them to consistently serialize themselves the same way they deserialize themselves.

If the script (or template) is read from a `string` value, it will serialize it self to xcontent as a `string` value as well. Otherwise it will serialize as an object, holding only those fields that where configured in the first place.

Original commit: elastic/x-pack-elasticsearch@52a82e0bbe
2015-05-05 23:25:14 +02:00
Brian Murphy 75bdf7de0d [TEST] Change execute test watches to have cron schedules that trigger in 2099.
Original commit: elastic/x-pack-elasticsearch@dde1dbae86
2015-05-05 12:37:44 -04:00
Brian Murphy b5f5862146 Clean up Watch parsing to disallow arrays.
This change cleans up the watch parsing which had allowed arrays in actions as a hold over from when actions was an array.

Fixes elastic/elasticsearch#367

Original commit: elastic/x-pack-elasticsearch@3f4fb82124
2015-05-05 10:44:28 -04:00
Brian Murphy 2ff8e2fb4e [TEST] Fix ack watch test to not trigger while test is running.
Original commit: elastic/x-pack-elasticsearch@d0261b355d
2015-05-04 16:19:41 -04:00
Brian Murphy b0fd2f9f59 Relocate guava and owasp under org.elasticsearch.watcher.common/common.owasp.
This change relocates the shaded jars for guava and owasp under org.elasticsearch.watcher.common and org.elasticsearch.watcher.common.owasp respectively.

Original commit: elastic/x-pack-elasticsearch@0e2d6303e0
2015-05-04 12:59:34 -04:00
Brian Murphy e6c4b53bf8 Move email html body sanitization to the EmailTemplate.render
This change moves the sanitization of the html body of emails to render time instead of at message send time.
Move the sanitization code the the EmailTemplate class.
Add <hr> tag to allowed html tags.
Add global setting `watcher.actions.email.sanitize_html` that defaults to true. If this is set to false html sanitization will be disabled
Add documentation for sanitization disable setting.
Enhance email tests to verify that sanitization is happening at construction time.

Fixes elastic/elasticsearch#356

Original commit: elastic/x-pack-elasticsearch@282a2d85c2
2015-05-04 12:44:43 -04:00
uboness 5e9f747005 Enable inlined xcontent structure in http input body
The body of the http input may hold xcontent data (json/yaml). The problem is that today, the body only accepts text and writing
 escapted json in text makes it very hard to maintain.

 This commit introduces another field settings - `xbody`. Now, use `xbody` for inlined xcontent and `body` for any text content. When `xbody` is used, we keep around the xcontent type and the http request that is then executed will include a content-type header that will match the xcontent type of input.

Original commit: elastic/x-pack-elasticsearch@7210908064
2015-05-04 13:28:32 +02:00
Martijn van Groningen 1dcb61654c Merge pull request elastic/elasticsearch#361 from martijnvg/tests/fix/HistoryTemplateTimeMappingsTests
Fix HistoryTemplateTimeMappingsTests

Original commit: elastic/x-pack-elasticsearch@834e84c57d
2015-05-04 13:24:19 +02:00
Martijn van Groningen e148b7e427 tests; added assertBusy(...) to get around the time issue in tests due to the async mapping update for fields introduced in tests.
Closes elastic/elasticsearch#351

Original commit: elastic/x-pack-elasticsearch@0df48aa7e8
2015-05-04 11:55:00 +02:00
uboness b63630496f Centralized all the date xcontent rendering
Original commit: elastic/x-pack-elasticsearch@08248c30cd
2015-05-04 04:53:47 +03:00
Brian Murphy d95e068a77 [TEST] Add html found during testing to cause problems for the sanitizer.
This change adds a html string that was found to cause problems for the sanitizer during testing.

Original commit: elastic/x-pack-elasticsearch@275e728674
2015-05-03 21:35:35 -04:00
Brian Murphy bc78dd36c9 Add and shade dependency on com.google.guava
This is needed for owasp to load the Sanitizers class at runtime.

Original commit: elastic/x-pack-elasticsearch@4ed5776509
2015-05-03 20:53:16 -04:00
Brian Murphy 0a545cb358 Change actual default of ignore_condition to false
Change default of ignore_codition to match docs as `false`.
Add test to verify defaults.

Fixes elastic/elasticsearch#348

Original commit: elastic/x-pack-elasticsearch@d740286a26
2015-05-03 19:33:04 -04:00
Brian Murphy 9bbaaa7588 [TEST] Mark HistoryTemplateTimeMappingsTests.testTimeFields awaitsfix.
Original commit: elastic/x-pack-elasticsearch@a9b59e23de
2015-05-03 13:30:59 -04:00
Brian Murphy 73e64292b0 [TEST] Add refresh to make sure watch record is in the index before checking the mappings.
Original commit: elastic/x-pack-elasticsearch@bd9b266af1
2015-05-03 12:53:14 -04:00
Brian Murphy 1d052e545d [TEST] Add logging to see why test is failing.
Log the metadata we get back to see why this test is failing on jenkins.

Original commit: elastic/x-pack-elasticsearch@7e2e4a5856
2015-05-03 12:36:23 -04:00
uboness dc6235a90a Updated watch_history template to better support aggregations
- added dynamic_templates to the index template that make sure not to analyze email addresses and http `path` and `host` under the `watch_execution` object. This will enabled building aggregations over these fields. Also make sure that different time fields in thw `watch_record` are mapped as date types.
- While at it changed the `watch_execution.input_result` mapping such that it's enabled and only disabled `watch_execution.input_result.payload` as different payloads from different sources may conflict with one another in terms of mappings
- Fixed a bug in the `EmailTemplate` building of the `reply_to` field.

- Also, added missing `execution_time` to the `watch_record` (under `watch_execution` object).

Closes elastic/elasticsearch#335

Original commit: elastic/x-pack-elasticsearch@dd28c70bca
2015-05-03 17:53:06 +03:00
Brian Murphy 7d8dc38c29 Change `meta` to metadata in the code.
The meta data field was known and `metadata` in the docs but the parsing logic expected `meta`.
This change unifies everything to use `metadata`. Also clean up Watch parsing to fail in a more consistent way when unknown fields are encountered.

Fixes elastic/elasticsearch#329

Original commit: elastic/x-pack-elasticsearch@a9013127d8
2015-05-03 07:44:35 -04:00
Brian Murphy 0d6fb1081a Hijack document APIs for .watches index
This change hijacks the document APIs for the .watches index and prevents direct access to these APIs via REST.
This hijacking is controlled by the "watcher.index.rest.direct_access" setting, this setting defaults to false which restricts access to the index. To allow direct access to the .watches index from the REST API set this setting to `true`
Copied json files from core es to test this feature and added comments to the json files indicating why they have been copied from es core.

Fixes elastic/elasticsearch#336

Original commit: elastic/x-pack-elasticsearch@22335750bd
2015-05-02 10:55:24 -04:00
Honza Král 86f0ea8d5a [API] don't include global parameter pretty
Original commit: elastic/x-pack-elasticsearch@15bc843bb1
2015-05-02 15:51:41 +02:00
Martijn van Groningen 299527c46a Serialize `input` field too.
Also added assertions to verify if all watch record not null field are expected to be there at parse time.

Original commit: elastic/x-pack-elasticsearch@42052ae02c
2015-05-02 11:31:52 +02:00
Martijn van Groningen 3b652d1772 test: fix failing tests
Original commit: elastic/x-pack-elasticsearch@3c35a828a4
2015-05-02 02:22:37 +02:00
Martijn van Groningen 31f039d1bd Make sure we skip over watches and watch records that are malformed during the Watcher starting phase.
Original commit: elastic/x-pack-elasticsearch@8bc37cdcc3
2015-05-02 01:45:21 +02:00
uboness c8a986cab7 Added a timeout to all watch write operations
The problem we have today is that since all write operations on a watch are blocked on the watch lock, if a watch is executing for a long time (for whatever reason) the action will simply block and by that hold a `management` thread for potentially a long time. This impacts elasticsearch as a whole - if for example there's an infinite loop in the condition script, then you try to delete/ack/update the watch 5 times (the default size of the management thread pool in elasticsearch is 5). You effectively block all management threads in elasticsearch and from that moment on, the only thing left to do is kill the node.

This commit introduces a timeout for the tree write watch actions - `put`, `ack` and `delete`. It is set to 10 seconds by default, but you can customize it both in java and in the rest API using the standard `master_timeout` query parameter.

Updated the relevant docs.

Fixes elastic/elasticsearch#309

Original commit: elastic/x-pack-elasticsearch@03292db37b
2015-05-01 19:02:29 +03:00
Brian Murphy b57e887095 Disallow the use of SCAN search type
This change disallows the SCAN search type in our search requests used by search input and search transform.
Add tests for this and update the current tests to both detect SCAN and not use SCAN in valid tests.

Original commit: elastic/x-pack-elasticsearch@c9d61930c8
2015-05-01 11:12:37 -04:00
Brian Murphy 56e422b9e8 Unify SearchRequest construction between SearchTransform and SearchInput.
This is a step between the old and elastic/elasticsearch#184 this change unifies the logic that generates the search requests from prototype for SearchTransforms and SearchInputs.
This change also adds the `executed_request` field to the `SearchTransform.Result` that was missing.
Add tests for SearchTransform.Result parsing and all different templating options for SearchTransform along with testing handling of search_type in SearchTransform which was being missed previously.

Fixes elastic/elasticsearch#311

Original commit: elastic/x-pack-elasticsearch@26e68576f5
2015-05-01 10:25:10 -04:00
Martijn van Groningen 118389bc06 shield: The rest layer can never use the Guice injected WatcherClient, but must always use the provided WatcherClient instance from the WatcherRestHandler base class.
Closes elastic/elasticsearch#316
Closes elastic/elasticsearch#317

Original commit: elastic/x-pack-elasticsearch@82df644895
2015-05-01 15:42:02 +02:00
Martijn van Groningen f66fc7575d test: moved all tests over to use source builders.
Original commit: elastic/x-pack-elasticsearch@e13e5b8b43
2015-05-01 11:54:56 +02:00
Brian Murphy 9df97fe845 Report the actual interval that was sent if less than 1000ms.
This change reports the actual interval that was used instead of just reporting the ms of the interval if the interval was less than 1000ms.

Fixes elastic/elasticsearch#301

Original commit: elastic/x-pack-elasticsearch@d625c952fe
2015-04-30 19:03:36 -04:00
Brian Murphy d0d646aa4c Remove default of 5m refresh on .watches and .watch_history* indices.
This default is far too high for normal use. It makes it very hard to build on top of watcher using the search APIs.

Fixes elastic/elasticsearch#279

Original commit: elastic/x-pack-elasticsearch@4e22d5c19e
2015-04-30 19:01:04 -04:00
Brian Murphy dad012fc2c Disable indexing of the WatchRecord.watch_exection.input_result
This input_result may contain different types for the same field names. This will cause mapping failures when we try to write the updated watch record.
This change disables this field and adds a test in the `ManualExecutionTests` to test this case.

Fixes elastic/elasticsearch#307

Original commit: elastic/x-pack-elasticsearch@5340761343
2015-04-30 18:05:16 -04:00
Brian Murphy 1819dc97f4 Throw exception when trying to use the default account but no accounts are configured.
This change throws an exception to let the user know that they are trying to send an email using the default account
but no accounts have been configured.

Fixes elastic/elasticsearch#303

Original commit: elastic/x-pack-elasticsearch@3e68bddf24
2015-04-30 16:45:36 -04:00
Brian Murphy ba3037f5fe When parsing a http request port is a required field.
The parser for http request had a bug and was accepting requests that did not specify a port.
This changes this.

Fixes elastic/elasticsearch#299

Original commit: elastic/x-pack-elasticsearch@e0eafe3787
2015-04-30 12:23:03 -04:00
uboness 67d2a39f2c [fix] fixed shield integration
- updated Shield version to 1.2.1 (final)
- Not all requests sent via the client proxy where set with the `__watcher_user`

Fixes elastic/elasticsearch#283

Original commit: elastic/x-pack-elasticsearch@3a36bd4d55
2015-04-30 02:08:27 +02:00
Martijn van Groningen 19d1e32453 restore merge mistake.
Original commit: elastic/x-pack-elasticsearch@b6e7d0ad60
2015-04-30 01:42:40 +02:00
Brian Murphy 274b01998c Do script compilation validation at parse time.
This change will attempt to compile scripts at construction time and throw an exception if a script is bad or missing.
With this change the script is compiled at Executable ctor and then the compiled script is used to get the executable.

Fixes elastic/elasticsearch#262

Original commit: elastic/x-pack-elasticsearch@b49c9debf2
2015-04-29 14:29:48 -04:00
Martijn van Groningen 5c3d94076e Made the `ticker` engine the default engine.
Original commit: elastic/x-pack-elasticsearch@f528bd5b8a
2015-04-29 20:17:35 +02:00
Martijn van Groningen 02a1c84c72 Fixed NPE when adding same job several times.
Closes elastic/elasticsearch#277

Original commit: elastic/x-pack-elasticsearch@42831113da
2015-04-29 20:17:35 +02:00
uboness 47a08171bd [cleanup] removed redundant comment in WatcherVersion
Original commit: elastic/x-pack-elasticsearch@b87b2d9bbc
2015-04-29 18:12:01 +02:00
uboness 20059d353a fixed the watcher info API
To return the right build info & version

Fixes elastic/elasticsearch#258

Original commit: elastic/x-pack-elasticsearch@53f5189978
2015-04-29 18:03:21 +02:00
Martijn van Groningen 7f24f229e2 http client: Only read response body if response code < 400.
The jdk http client will not allow reading the body if the response code >= 400.

Relates to elastic/elasticsearch#271

Original commit: elastic/x-pack-elasticsearch@08836f92ae
2015-04-29 17:39:22 +02:00
Martijn van Groningen 04f4a09feb schedules: Don't allow schedules with negative interval.
Closes elastic/elasticsearch#270

Original commit: elastic/x-pack-elasticsearch@a24941683b
2015-04-29 15:40:39 +02:00
Martijn van Groningen f4d0dc4955 http client: Don't append '?' to the url if there are no query string options defined.
Closes elastic/elasticsearch#271

Original commit: elastic/x-pack-elasticsearch@63487fad87
2015-04-29 15:07:48 +02:00
Martijn van Groningen 5bc491d359 execution service: Don't try to send empty bulk requests
Closes elastic/elasticsearch#264

Original commit: elastic/x-pack-elasticsearch@f636bfd956
2015-04-29 15:06:16 +02:00
Martijn van Groningen e438c51555 execution service: Ignore watch records that where deleted while awaiting for execution.
Closes elastic/elasticsearch#268

Original commit: elastic/x-pack-elasticsearch@dbb7160a67
2015-04-29 15:02:49 +02:00
Brian Murphy 42d49d54ef Merge pull request elastic/elasticsearch#265 from GaelTadh/api_namespace_changes
Change watcher REST API call namespace to be watcher.

Original commit: elastic/x-pack-elasticsearch@7a80eb522b
2015-04-28 23:41:03 -04:00
Martijn van Groningen 66c492192c test: fix test bug
Original commit: elastic/x-pack-elasticsearch@d747eadf2f
2015-04-29 01:28:38 +02:00
Brian Murphy 94bea8f83a Change the watch specific actions to reflect that they operate on watches.
This change changes the watch specific actions put/get/delete/execute/ack to be watcher.<action>_watch.

Original commit: elastic/x-pack-elasticsearch@86a91cb141
2015-04-28 19:08:28 -04:00