Commit Graph

5345 Commits

Author SHA1 Message Date
Martijn van Groningen f8f7454130 [TEST] Actually perform the intended close...
Original commit: elastic/x-pack-elasticsearch@1333d78509
2017-04-12 09:23:48 +02:00
Martijn van Groningen 272d1b269e Muted test.
Original commit: elastic/x-pack-elasticsearch@a4d8a72023
2017-04-12 09:08:06 +02:00
lcawley 412bb63383 [DOCS] Remove data type formatting from API pages
Original commit: elastic/x-pack-elasticsearch@fb06ece3f0
2017-04-11 20:17:29 -07:00
lcawley 1a6f813d5a [DOCS] Update all ML API examples with latest build output
Original commit: elastic/x-pack-elasticsearch@f9fa3b813a
2017-04-11 20:15:50 -07:00
lcawley 298ee9f554 [DOCS] Add snapshot API examples
Original commit: elastic/x-pack-elasticsearch@8928c3c1be
2017-04-11 14:43:17 -07:00
lcawley a0771019a5 [DOCS] Remove redundant batch_span and period from job config
Original commit: elastic/x-pack-elasticsearch@95e162eaa9
2017-04-11 13:30:03 -07:00
Martijn van Groningen 4f34af20ba Muted test.
Original commit: elastic/x-pack-elasticsearch@da69d049ad
2017-04-11 20:51:52 +02:00
Igor Motov 428af93f7b Persistent Tasks: switch from long task ids to string task ids (elastic/x-pack-elasticsearch#1035)
This commit switches from long persistent task ids to caller-supplied string persistent task ids.

Original commit: elastic/x-pack-elasticsearch@2dff985df7
2017-04-11 12:24:54 -04:00
Simon Willnauer a7dfbcd2cb [TEST] Use SecurityClusterClientYamlTestCase to ensure mappings and templates are present
Original commit: elastic/x-pack-elasticsearch@742633c93c
2017-04-11 18:11:31 +02:00
lcawley a1bf6247a8 [DOCS] Add ML API results examples
Original commit: elastic/x-pack-elasticsearch@60a21763eb
2017-04-11 09:00:15 -07:00
Zachary Tong 6f643ffba5 [ML] If snapshot list is empty, return after signaling failure
Original commit: elastic/x-pack-elasticsearch@a89f88da11
2017-04-11 11:37:25 -04:00
Alexander Reelsen 71852c7215 Security: Add watcher_user and watcher_admin role (elastic/x-pack-elasticsearch#983)
This built-in watcher_admin role is able to execute all watcher actions,
read the watch history indices and read the watches index
index. The watcher_user role allows to GET a watch and to get the stats and thats it.

relates elastic/x-pack-elasticsearch#978

Original commit: elastic/x-pack-elasticsearch@11b33a413b
2017-04-11 16:28:55 +01:00
Dimitrios Athanasiou a5f0a74b67 [ML] Remove unused imports in GetDatafeedsStatsAction
Original commit: elastic/x-pack-elasticsearch@83abaae58e
2017-04-11 16:26:13 +01:00
Ali Beyad c468a4d3a3 [TEST] Security index template/mappings check on tests (elastic/x-pack-elasticsearch#1031)
Extends the security index check that ensure the template/mappings
are up-to-date to other security integration tests that depend on this
check.

relates elastic/x-pack-elasticsearch#794

Original commit: elastic/x-pack-elasticsearch@ec8e5b37bd
2017-04-11 11:22:41 -04:00
Martijn van Groningen 0e508d1bfb [TEST] Adjusted log levels to better track failure elastic/x-pack-elasticsearch#1048
Original commit: elastic/x-pack-elasticsearch@8f8b9fe17e
2017-04-11 16:58:49 +02:00
Dimitris Athanasiou 3fa43093eb [ML] Improve handling of job errors from datafeed (elastic/x-pack-elasticsearch#1049)
- stops the datafeed when post/flush throw a conflict exception.
A conflict exception signifies the job state is not opened, thus
we are better off stopping the datafeed.
- handles flushing the job the same way as posting to the job.

relates elastic/x-pack-elasticsearch#855

Original commit: elastic/x-pack-elasticsearch@49a54912c2
2017-04-11 15:55:05 +01:00
David Roberts 9525cb4784 [ML] Replace UTF-8 infinity symbol with ASCII word (elastic/x-pack-elasticsearch#1041)
Makes the log more readable in editors not set to UTF-8.
Customers may well be in this situation on Linux/Windows.

Original commit: elastic/x-pack-elasticsearch@4e59fc90cf
2017-04-11 15:25:50 +01:00
David Kyle 78a3c32ec4 [ML] Reduce frequency of data counts log message (elastic/x-pack-elasticsearch#1030)
* Reduce data counts log message frequency

* Start logging at 10,000 records

Original commit: elastic/x-pack-elasticsearch@af6c791b6e
2017-04-11 15:12:41 +01:00
David Kyle f11df2c0c7 [ML] Give the jobs in index_layout.yaml more meaningful names (elastic/x-pack-elasticsearch#1047)
For debugging the CI failures

Original commit: elastic/x-pack-elasticsearch@5f6a3d1c6d
2017-04-11 14:58:48 +01:00
Martijn van Groningen e1aac25f2d [TEST] Ensure all nodes indicate job has been stopped
Original commit: elastic/x-pack-elasticsearch@39184b6c69
2017-04-11 15:14:41 +02:00
jaymode cb735b5f69 Test: add missng action cluster:monitor/remote/info
Original commit: elastic/x-pack-elasticsearch@11429d81a7
2017-04-11 09:04:34 -04:00
Tanguy Leroux 1243c64671 [TEST] Remove log traces and change how LocalExporterTests stops
The commit changes how LocalExporterTests stops: it now uses the
node_stats document collected on each node and check if it's older
than a given number of seconds (10). It also removes log traces.

Original commit: elastic/x-pack-elasticsearch@0384690b41
2017-04-11 14:50:44 +02:00
Simon Willnauer 9c542173b7 [TEST] Make test more robust against slow connection establishing
Original commit: elastic/x-pack-elasticsearch@6f918a44c9
2017-04-11 14:43:44 +02:00
Dimitris Athanasiou 4da1c5b9dc [ML] Remove batch_span and period from analysis config (elastic/x-pack-elasticsearch#1043)
relates elastic/x-pack-elasticsearch#1040

Original commit: elastic/x-pack-elasticsearch@aed5e9912c
2017-04-11 12:57:58 +01:00
Hendrik Muhs 7ef9a16f45 [ML] implement '_all' for stopping datafeeds (elastic/x-pack-elasticsearch#995)
Add a '_all' functionality for stopping ML datafeeds.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to stop all datafeeds at once where previously it was required to iterate over all feeds and do a explicit stop. This is part two of elastic/x-pack-elasticsearch#795, part one can be found in elastic/x-pack-elasticsearch#962 .

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@ed1eff83d5
2017-04-11 13:39:22 +02:00
Shaunak Kashyap 01a8f5a0b3 Document security privileges required for using Watcher (elastic/x-pack-elasticsearch#1012)
* Document security privileges required for using Watcher

* Fixing index names

* Language fixes

Original commit: elastic/x-pack-elasticsearch@56ef06d077
2017-04-11 04:22:45 -07:00
David Roberts d2c90e798f [ML] Make open/close job timeouts consistent (elastic/x-pack-elasticsearch#1039)
Before this change the persistent task operations related to opening
and closing jobs would time out a long time before the operations
related to native processes.

Original commit: elastic/x-pack-elasticsearch@23076b773b
2017-04-11 11:45:48 +01:00
Dimitrios Athanasiou 1760677b0e [ML] Remove redundant logging
Relates elastic/x-pack-elasticsearch#1027

Original commit: elastic/x-pack-elasticsearch@ddbe390e6f
2017-04-11 10:44:00 +01:00
Simon Willnauer 617c3ead5c Add tests for _remote/info API (elastic/x-pack-elasticsearch#1009)
Relates to elastic/elasticsearch#23925
Depends on elastic/elasticsearch#23969

Original commit: elastic/x-pack-elasticsearch@d1e8754a57
2017-04-11 11:24:22 +02:00
Clinton Gormley d4d505fb97 Added docs for making cross cluster search work with security
relates elastic/x-pack-elasticsearch#868

Original commit: elastic/x-pack-elasticsearch@0292fbb5d3
2017-04-11 10:08:12 +02:00
Colin Goodheart-Smithe 29c64d83e1 Removes version 2.x constants from Version (elastic/x-pack-elasticsearch#1022)
Original commit: elastic/x-pack-elasticsearch@39ef909aec
2017-04-11 08:30:59 +01:00
Tim Vernum 8840042751 Only log LDAP errors if authentication failed overall (elastic/x-pack-elasticsearch#960)
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"

Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request. 
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.

This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
  
Closes: elastic/x-pack-elasticsearch#887

Original commit: elastic/x-pack-elasticsearch@b81b363729
2017-04-11 09:37:21 +10:00
Zachary Tong 43f1fb2bb1 [ML] Correctly parse stream of SMILE documents in JsonDataToProcessWriter (elastic/x-pack-elasticsearch#982)
The PR detects if SMILE is being provided, then correctly slices the stream such that each document is parsed individually. This is required because jackson's SMILE parser is stricter than it's JSON parser and will stop parsing when it hits a streamSeparator (unlike JSON, which will eagerly try to find more objects to parse).

Removes the forced-headers from the various REST tests.

relates elastic/x-pack-elasticsearch#642 

Original commit: elastic/x-pack-elasticsearch@c0e97cd545
2017-04-10 17:55:15 -04:00
Igor Motov 49223a8782 Persistent Tasks: remove listener from PersistentTasksExecutor#nodeOperation (elastic/x-pack-elasticsearch#1032)
Instead of having a separate listener for indicating that the current task is finished, this commit is switching to use allocated object itself.

Original commit: elastic/x-pack-elasticsearch@7ad5362121
2017-04-10 17:32:30 -04:00
Martijn van Groningen 018a3d197d [ML] Index job and datafeed assignment notifications via a separate cluster state listener.
`PersistentTasksExecutor#getAssignment(...)` should be a cheap and side-effect free method,
but in case of `OpenJobPersistentTasksExecutor` and `StartDatafeedPersistentTasksExecutor` before this change it would index a document each time `getAssignment(...)` was invoked

Original commit: elastic/x-pack-elasticsearch@5ca5890baf
2017-04-10 20:20:48 +02:00
Martijn van Groningen ecc19d9948 [TEST] Fixed typos in @TestLogging annotations
Original commit: elastic/x-pack-elasticsearch@3152765d4c
2017-04-10 20:13:50 +02:00
Martijn van Groningen 9201a5d640 [TEST] Changed test log levels
Original commit: elastic/x-pack-elasticsearch@73bd1251fa
2017-04-10 19:48:56 +02:00
Dimitris Athanasiou 1e1b5405b3 [ML] Also chunk aggregated datafeed by default (elastic/x-pack-elasticsearch#999)
The change applies chunking by default on aggregated datafeeds.
The chunking is set to a manual mode with time_span being
1000 histogram buckets.

The motivation for the change is two-fold:

1. It helps to avoid memory pressure/blowing.
Users may perform a lookback on a very long period of time. In that
case, we may hold a search response for all that time which could
include too many buckets. By chunking, we avoid that situation
as we know we'll only keep results for 1000 buckets at a time.

2. It makes cancellation more responsive.
In elastic/x-pack-elasticsearch#862 we made the processing of a search response cancellable in a
responsive manner. However, the search phase cannot be cancelled at
the moment. Chunking makes the search phase shorter, which will
result to a better user experience when they stop an aggregated
datafeed.

Also note the change sets the default chunking_config on datafeed
creation so the setting is no longer hidden.

Relates to elastic/x-pack-elasticsearch#803

Original commit: elastic/x-pack-elasticsearch@ae8f120f5f
2017-04-10 18:20:48 +01:00
Dimitris Athanasiou 0b6ac175da [ML] Check datafeed can be assigned before creating task (elastic/x-pack-elasticsearch#1027)
When a datafeed task is created but it cannot be assigned the task
has a null status. This means _stats report it as stopped, however
deleting it fails. In addition, it's a better experience to error
the start datafeed request all together and give the user the chance
to fix his data indices.

This change fails a datafeed-start if it cannot be assigned.

relates elastic/x-pack-elasticsearch#1018

Original commit: elastic/x-pack-elasticsearch@532288fda0
2017-04-10 18:19:46 +01:00
Igor Motov 105ba131e0 Persistent Tasks: remove retries on notification failures (elastic/x-pack-elasticsearch#977)
Retries should be already handled by TransportMasterNodeAction, there is no need to introduce another retry layer in Persistent Tasks code.

Original commit: elastic/x-pack-elasticsearch@967ac7f7fa
2017-04-10 12:26:09 -04:00
Lisa Cawley 90575b18f4 [DOCS] Add ML data feed API examples (elastic/x-pack-elasticsearch#1016)
* [DOCS] Added examples for all ML job APIs

* [DOCS] Add ML datafeed API examples

Original commit: elastic/x-pack-elasticsearch@9634356371
2017-04-10 09:06:34 -07:00
David Kyle 00bc35cf9f [ML] Consistently throw missing resource exception on all endpoints if the job id is not known. (elastic/x-pack-elasticsearch#1024)
Original commit: elastic/x-pack-elasticsearch@6676a03599
2017-04-10 15:23:08 +01:00
Tanguy Leroux 310b85083d [Test] Change how LocalExporterTests stops
This commit changes how LocalExporterTests stops the monitoring
components: it first stops the monitoring service (but keeps the
local exporter enabled), deletes and checks if monitoring indices
are recreated, and then disables the local exporter.

Original commit: elastic/x-pack-elasticsearch@4c4809a660
2017-04-10 15:14:49 +02:00
Martijn van Groningen 454d6b3390 [ML] Reintroduced the closing job state
Closing a job may take a while. In the meantime it is possible to start a datafeed, because before this change the job state remained OPENED.
With this change when the executor node receives the close job request, it will first set the status to CLOSING and after that closes the job (closing autodetect process, etc.).

relates elastic/x-pack-elasticsearch#990

Original commit: elastic/x-pack-elasticsearch@d8d89c0756
2017-04-10 15:08:50 +02:00
Tanguy Leroux 6396edc6a7 [Test] Add Monitoring Bulk API REST tests with privileges (elastic/x-pack-elasticsearch#968)
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.

Original commit: elastic/x-pack-elasticsearch@f1665815c2
2017-04-10 15:08:19 +02:00
javanna 07a99c4792 [TEST] exclude num_reduce_phases from the SearchTransformTests comparisons
Original commit: elastic/x-pack-elasticsearch@d3a5c7e86b
2017-04-10 14:21:38 +02:00
David Roberts e8337344a6 [ML] Include C++ 3rd party notices in X-Pack combined NOTICES file (elastic/x-pack-elasticsearch#953)
relates elastic/x-pack-elasticsearch#620

Original commit: elastic/x-pack-elasticsearch@5d73bd340c
2017-04-10 12:40:07 +01:00
Martijn van Groningen 63c04ef19d [ML] Change close job api to no longer extend from TransportJobTaskAction.
The execution has diverged too much from post data, flush and update process apis, since the close all jobs have been added.
The logic is now easier to understand as it exist in a single source file instead of in both CloseJobAction and TransportJobTaskAction.

Original commit: elastic/x-pack-elasticsearch@daf5fabad5
2017-04-10 13:33:12 +02:00
Luca Cavanna cb07e93fde Adapt to getProperty move out of MultiBucketsAggregation.Bucket (elastic/x-pack-elasticsearch#1015)
Adapt to getProperty move out of MultiBucketsAggregation.Bucket

Original commit: elastic/x-pack-elasticsearch@3e8ca753cd
2017-04-10 13:28:37 +02:00
Tim Vernum 18835fe4df Expand debug logging for authentication / realms (elastic/x-pack-elasticsearch#641)
Users currently have difficulty diagnosing authentication failures.
Some logging messages mislead them, and in other cases there are unexpected behaviours that are not logged at all.
These additional DEBUG log messages and change some existing messages in an attempt to alleviate that problem.

Original commit: elastic/x-pack-elasticsearch@c6ea98b038
2017-04-10 21:21:40 +10:00