Commit Graph

5148 Commits

Author SHA1 Message Date
Martijn van Groningen 93a2a567cb [ML] Cut open job and start datafeed apis to use persistent task's assignment explanation.
Also included assignment explanation to both job and datafeed stats apis and
included executor node to datafeed stats api

Original commit: elastic/x-pack-elasticsearch@783bc77ef6
2017-02-28 10:38:51 +01:00
David Kyle 69880373fc Remove the ml_messages.properties file (elastic/x-pack-elasticsearch#657)
String constants are no in Messages

Original commit: elastic/x-pack-elasticsearch@ab8af83e10
2017-02-28 09:33:47 +00:00
Dimitris Athanasiou 2d27e8ad81 [ML] Warn when lookback only datafeed retrieved no data (elastic/x-pack-elasticsearch#651)
When the datafeed indexes are concrete indexes, the search
will throw IndexNotFoundException, thus a notification is being
generated. However, when the indexes are patterns, the search
does not fail.

This commit adds a warning when a lookback-only datafeed
retrieves no data as it is certain that the user misconfigured
something and it's useful to make him aware of it.

Original commit: elastic/x-pack-elasticsearch@e3490ea978
2017-02-27 14:18:07 +00:00
javanna adc9808eff Convert watcher search source to json
If any format other than json is sent, it should get converted to json, as watcher works under the assumption that the script is a template, hence json.

Original commit: elastic/x-pack-elasticsearch@554bcd306b
2017-02-27 12:28:04 +01:00
javanna 39ccc67a7b disable content-type randomization for some ml tests
post_data and get_model seem to support json only

Original commit: elastic/x-pack-elasticsearch@683f998228
2017-02-27 12:28:04 +01:00
javanna 33ccc3bd6c adapt to ObjectPath changes to support binary formats
Original commit: elastic/x-pack-elasticsearch@11782418e9
2017-02-27 12:28:04 +01:00
Martijn van Groningen 212098fc7c [TEST] Attempt to stabilize distributed ml tests.
Relates to elastic/x-pack-elasticsearch#653

Original commit: elastic/x-pack-elasticsearch@e4919ab2b4
2017-02-27 11:11:19 +01:00
David Roberts e2f5715191 [TEST] Fix one last test that was missed in elastic/x-pack-elasticsearch#649
Original commit: elastic/x-pack-elasticsearch@1039197afd
2017-02-27 09:55:49 +00:00
David Roberts 2ca1ddf017 [ML] Tell native controller process to stop during graceful shutdown (elastic/x-pack-elasticsearch#644)
Fixes elastic/prelert-legacy#803

On Windows the JVM hangs during shutdown if the native controller process
is still running. A thread dump suggests this is because of a finalizer being
blocked. We can avoid the problem by explicitly telling the controller to exit
when the node is stopped. Although the hang is Windows-specific, it doesn't
hurt to do this on *nix too.

Original commit: elastic/x-pack-elasticsearch@6fa1c2b966
2017-02-27 08:52:03 +00:00
Colin Goodheart-Smithe 4a81c85f71 Reapplying: Adds cast hint for eclipse
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners

This fix was mistakenly removed in another commit

Original commit: elastic/x-pack-elasticsearch@6305456d5a
2017-02-27 08:43:27 +00:00
Dimitris Athanasiou 380f6d9bff [ML] Add datafeed preview end-point (elastic/x-pack-elasticsearch#645)
Adds a preview end-point: `/_xpack/ml/datafeeds/{datafeed_id}/_preview`

The endpoint returns an array with the JSON records that will reach
writer to the c++ process. Thus, the preview can be used to verify
that a datafeed is configured correctly with regard to itself and
its corresponding job.

Original commit: elastic/x-pack-elasticsearch@1e0e9c906d
2017-02-25 10:17:06 +00:00
Dimitris Athanasiou d49930040e [TEST] Disable autodetect_process in tests not needing it (elastic/x-pack-elasticsearch#649)
Original commit: elastic/x-pack-elasticsearch@38897c0040
2017-02-24 18:57:33 +00:00
David Kyle 08b7cb6bca [ML] Refactor Auditor (elastic/x-pack-elasticsearch#631)
* Refactor Auditor
* Inject Auditor


Original commit: elastic/x-pack-elasticsearch@d9efe5fcef
2017-02-24 15:56:38 +00:00
Martijn van Groningen a189e1b759 [TEST] Added more failure scenarios
Original commit: elastic/x-pack-elasticsearch@a27134323e
2017-02-24 16:35:28 +01:00
David Kyle 8a3706f395 [ML] Influencers come after analysis fields (elastic/x-pack-elasticsearch#646)
Original commit: elastic/x-pack-elasticsearch@f37835b140
2017-02-24 15:12:35 +00:00
Martijn van Groningen 8ecdf3b288 [TEST] Attempt to reduce `MockPageCacheRecycler` assertion failures by waiting until there are no active recoveries.
Original commit: elastic/x-pack-elasticsearch@1b142f29bc
2017-02-24 15:23:02 +01:00
Martijn van Groningen 2c5e32f3d3 [ML] Removed workaround that task status named writable needed to have the same name as action name.
Original commit: elastic/x-pack-elasticsearch@99d9d80073
2017-02-24 14:32:53 +01:00
Martijn van Groningen cd4e543479 [ML] Tweaked starting datafeed logic, so that it will not start when job hasn't started.
[TEST] Unmuted ml full cluster restart test.

Original commit: elastic/x-pack-elasticsearch@c8cd9870c9
2017-02-24 13:47:44 +01:00
David Kyle f67d8c3746 [ML] Fix intermittent test failure (occasionally deleted an index we don’t care about)
Original commit: elastic/x-pack-elasticsearch@56c7bdb5b9
2017-02-24 12:20:47 +00:00
David Kyle a09e8a260a [ML] Use Index Templates (elastic/x-pack-elasticsearch#622)
* [ML] Index template for notifications, meta and state indices

* Add ignore_unavailable option to searches so missing indices don’t throw errors

* Test for job existence before returning results

* Template mapping for results indices

* Fix tests

* Reinstate merge conflict dropped code

* Fix and tidy up JobProviderTests

Tests had badly named & duplicated tests

* Don’t check if the job indices already exist when allocating

Indices are created on demand from templates

* No need to create indices in initialisation service

Using templates now

* Remove unused code and add comment

Original commit: elastic/x-pack-elasticsearch@1ba115d4fc
2017-02-24 10:20:07 +00:00
Colin Goodheart-Smithe 7f6edbdd4b Adds cast hint for eclipse
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners

Original commit: elastic/x-pack-elasticsearch@a7aff45571
2017-02-24 08:38:33 +00:00
Igor Motov 615dd0f240 Request and Status in Persistent Tasks should be serialized using their writable names
Refactors xcontent serialization of Request and Status to use their writable names instead of action name. That simplifies the parsing logic, allows reuse of the same status object for multiple actions and is consistent with how named objects in xcontent are used.

Original commit: elastic/x-pack-elasticsearch@3d059ea952
2017-02-23 18:55:38 -05:00
Martijn van Groningen 9f48a1f677 [ML] Added the node the job is running on to the response of the job stats api.
Original commit: elastic/x-pack-elasticsearch@d92149ca59
2017-02-23 20:48:40 +01:00
Jay Modi 405230c308 Do not add null listeners to the DnRoleMapper (elastic/x-pack-elasticsearch#636)
The DnRoleMapper was changed when moving to asynchronous authentication, which introduced a regression that leads to a
NPE being thrown on file refresh. The cause is the addition of a null listener from the LdapRealm upon construction.

This change removes the listener constructor argument as it was always null or a empty runnable and adds a check in the
addListener method to ensure the listener is not null. Finally, the DnRoleMapperTests had a bug where the temporary
filenames did not have the correct suffix and tests were not actually testing what was intended.

relates elastic/x-pack-elasticsearch#608

Original commit: elastic/x-pack-elasticsearch@f47f258590
2017-02-23 11:16:14 -07:00
Ali Beyad 4a001706e7 Simplifies security index state changes and template/mapping updates (elastic/x-pack-elasticsearch#551)
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.

This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.

This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.

Original commit: elastic/x-pack-elasticsearch@b31d144597
2017-02-23 13:05:39 -05:00
jaymode e61f87d5ff Test: do not use the same name twice
In rare cases, the CertificateToolTests#testPromptingForInstanceInformation could try to use the same name multiple
times. This causes the test to fail with a message for an unknown prompt. This commit ensures no duplicates are added.

relates elastic/x-pack-elasticsearch#597

Original commit: elastic/x-pack-elasticsearch@ab8bb7ad50
2017-02-23 12:34:06 -05:00
Igor Motov db88cc458d Add persistent task assignment explanations.
This commit allows persistent actions to indicate why a task was or wasn't assigned to a certain node.

Original commit: elastic/x-pack-elasticsearch@cdacf9b10f
2017-02-23 11:22:08 -05:00
Martijn van Groningen d87926ab86 [ML] Never stop the datafeeder when no data can be found
Changed `ProblemTracker#updateEmptyDataCount(boolean)` into `ProblemTracker#reportEmptyDataCount()`.
and added `ProblemTracker#reportNoneEmptyCount()` to reset the empty count counter.

Original commit: elastic/x-pack-elasticsearch@c1a44d6fd3
2017-02-23 16:32:08 +01:00
Martijn van Groningen e7d56e92f8 [Ml] Rename `useNativeProcess` setting to `xpack.ml.autodetect_process` and un-deprecate it.
Original commit: elastic/x-pack-elasticsearch@eb2143e6f0
2017-02-23 15:29:44 +01:00
Dimitris Athanasiou 412d5e35b1 [ML] model_debug_config: add enabled and remove bounds_percentile (elastic/x-pack-elasticsearch#627)
This allows it to be disabled/enabled via job updates.
It also simplifies it by removing bounds_percentile as it currently
rarely makes sense to set the bounds to another value than 95.0.

Original commit: elastic/x-pack-elasticsearch@c27fce2d86
2017-02-23 14:11:23 +00:00
David Roberts 33b10970ee [ML] Temporary workaround to make some non-ml integration tests work (elastic/x-pack-elasticsearch#630)
This will be replaced with a better solution when elastic/x-pack-elasticsearch#566 is implemented

Original commit: elastic/x-pack-elasticsearch@bf0d9e0176
2017-02-23 11:13:58 +00:00
Dimitris Athanasiou d3d73ca095 [ML] Remove data that expired their respective retention period (elastic/x-pack-elasticsearch#610)
This PR introduces a service component that is scheduled to run
once per day, around 30 minutes past midnight, local time.
Listeners can be subscribed to the service in order to get notified
when the next service trigger occurs so that they can execute
management tasks.

Currently, there are 2 such listeners: one is performing removal
of results and the other of model snapshots that have expired
their retention period.

Finally, note that the service is only installed on the master
node to ensure there is only one running on the cluster.

Original commit: elastic/x-pack-elasticsearch@c8edb6016a
2017-02-23 11:06:14 +00:00
Martijn van Groningen a03e9d36c2 [TEST] mute test
Original commit: elastic/x-pack-elasticsearch@3989e5a074
2017-02-23 11:38:44 +01:00
Martijn van Groningen b40d5dde9b [ML] Before selecting an executor node for running job task, make sure that all required indices exist and are accessible.
Original commit: elastic/x-pack-elasticsearch@5459a0d5f2
2017-02-23 10:55:16 +01:00
Alexander Reelsen 43a09a2fa7 Watcher: Add license files (elastic/x-pack-elasticsearch#619)
Add license, notice and SHA sums for watcher compile dependencies
(excluding test deps).

Relates elastic/x-pack-elasticsearch#607

Original commit: elastic/x-pack-elasticsearch@06ce510436
2017-02-23 10:24:16 +01:00
Martijn van Groningen 2eba41cfac [TEST] Fix mocking bug
Original commit: elastic/x-pack-elasticsearch@61268c1180
2017-02-23 09:59:45 +01:00
Martijn van Groningen 6195000b8a [TEST] Change ml cleanup workaround.
Original commit: elastic/x-pack-elasticsearch@793d6cfb78
2017-02-23 09:08:10 +01:00
Tim Vernum 9f6285f362 [TEST] Waiting longer for native realm migration
Let LicensingTests.testNativeRealmMigratorWorksUnderBasicLicense wait for 3s (instead of 1s) as it is occasionally failing.
Also added logging so that if/when it does fail we can see how long we were waiting and what was taking place.

Original commit: elastic/x-pack-elasticsearch@22882ec412
2017-02-23 14:15:17 +11:00
Tim Vernum 672819f038 LICENSE files for security (elastic/x-pack-elasticsearch#614)
* License information for unboundid-ldapsdk
   From: https://docs.ldap.com/ldap-sdk/docs/LICENSE-UnboundID-LDAPSDK.txt

* License information for Bouncy Castle
   From:  java -cp ~/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.55/bcprov-jdk15on-1.55.jar org.bouncycastle.LICENSE


Original commit: elastic/x-pack-elasticsearch@b32d5aade3
2017-02-23 10:43:34 +11:00
Ryan Ernst bca7d87ac4 Rename threadpool time method
See elastic/elasticsearch#23175

Original commit: elastic/x-pack-elasticsearch@0e185c0a44
2017-02-22 15:11:42 -08:00
Dimitris Athanasiou eda2f15b44 [ML] Remove deprecated use of TransportDeleteAction (elastic/x-pack-elasticsearch#628)
Original commit: elastic/x-pack-elasticsearch@b5fdb32a0c
2017-02-22 22:14:32 +00:00
Martijn van Groningen 783a2cf01b [TEST] Added more test logging
Original commit: elastic/x-pack-elasticsearch@d68a1e3510
2017-02-22 21:47:01 +01:00
Ryan Ernst 2cd3979654 Merge pull request elastic/x-pack-elasticsearch#617 from rjernst/integ_test_order
Build: Convert integ test dsl to new split cluster/runner dsl

Original commit: elastic/x-pack-elasticsearch@454edd48d1
2017-02-22 12:43:25 -08:00
Martijn van Groningen 124fe0ad04 [ML] Fix line length
Original commit: elastic/x-pack-elasticsearch@2fba2434b8
2017-02-22 21:37:25 +01:00
polyfractal 4041545cfa [ML] Put back check for inconsistent job id
Lost during the refactor, this checks to make sure an ID in both the URL and the
body match.

Original commit: elastic/x-pack-elasticsearch@ef96275c99
2017-02-22 15:21:03 -05:00
Martijn van Groningen 9da18a14c5 [ML] Set createTime if not specified in validate api too
Original commit: elastic/x-pack-elasticsearch@477fb0fc52
2017-02-22 21:03:55 +01:00
polyfractal f856a79ef3 Fix method signature in test
Original commit: elastic/x-pack-elasticsearch@4340952f77
2017-02-22 14:08:19 -05:00
Zachary Tong 41811efcf1 [ML] Job's CreateTime must be non-null (elastic/x-pack-elasticsearch#616)
Also simplifies Job.Builder to only have one build() method.

Original commit: elastic/x-pack-elasticsearch@50e4b4c210
2017-02-22 13:45:13 -05:00
Martijn van Groningen 9a5b410110 [TEST] Add test that verifies that machine learning works correctly after full cluster restart.
Added missing named xcontent entries, tweaked start datafeed validation logic and fixed status serialization issue that were uncovered by this test.

Original commit: elastic/x-pack-elasticsearch@8b3fb012df
2017-02-22 17:25:08 +01:00
Dimitrios Athanasiou 4bb57137a9 fix unit test
Original commit: elastic/x-pack-elasticsearch@af9276f02f
2017-02-22 14:42:39 +00:00