Commit Graph

5211 Commits

Author SHA1 Message Date
Alexander Reelsen e492b17c10 Watcher: Ensure all json builders use try-with-resources (elastic/x-pack-elasticsearch#650)
Some json builders in the codebase were not closed. even
though this is not needed for the BytesStreamOutput being used,
there is more closing logic in the jackson classes, which we
should not rely on, that those never change or are ok to not
close.

Original commit: elastic/x-pack-elasticsearch@05a43d80ff
2017-03-01 13:45:10 +01:00
David Kyle 4edfbe664f [ML] Delete unused AuditActivity class (elastic/x-pack-elasticsearch#672)
Original commit: elastic/x-pack-elasticsearch@185ceab4b4
2017-03-01 10:42:27 +00:00
David Roberts 094be3c28b [ML] Put C++ process/PID and file@line in square brackets (elastic/x-pack-elasticsearch#671)
This is the Elastic convention, and also makes it clearer where the
actual log message from the C++ starts

Original commit: elastic/x-pack-elasticsearch@dc9aeefb08
2017-03-01 09:33:30 +00:00
Martijn van Groningen ac93195407 [TEST] Don't install xpack plugin as the test doesn't rely on it initializes the ml classes it tests on its own.
Original commit: elastic/x-pack-elasticsearch@e505efcddb
2017-03-01 09:32:34 +01:00
Igor Motov 377c1ec2b4 Simplify names of PersistentTasks-related classes
PersistentTask -> NodePersistentTask
PersistentTasksInProgress -> PersistentTasks
PersistentTaskInProgress -> PersistentTask

Original commit: elastic/x-pack-elasticsearch@0947dbca56
2017-02-28 16:37:09 -07:00
Areek Zillur ec4de10ee2 Tribe node security tests with external clusters (elastic/x-pack-elasticsearch#606)
* Tribe node security tests with external clusters

This PR adds a qa module for security tests with tribe node
using external clusters. Existing SecurityTribeIT tests
have been ported to use external clusters with tribe setup
as a first step.

Currently the ports to the external clusters are passed to the
integration tests through system properties and external clusters
are built on test setup (the code for building external clusters is
copied from ESIntegTestCase). This is a WIP as we need a
more generic way to facilitate testing tribe setup with external
clusters. thoughts welcome.

* incorporate feedback

* update to master

Original commit: elastic/x-pack-elasticsearch@686887ca91
2017-02-28 18:36:53 -05:00
Martijn van Groningen 9d6c344520 [ML] Disable ml in case a node is a tribe node and
in case of tribe node client do not even bother to check for native process version, because it isn't running.

Original commit: elastic/x-pack-elasticsearch@e6fdd5e82a
2017-02-28 21:31:09 +01:00
Martijn van Groningen 6783f823a8 [ML] Change stop datafeed api delegate to node hosting datafeed task and execute cancel locally,
instead of only removing the persistent task from cluster state.

Original commit: elastic/x-pack-elasticsearch@3974b20827
2017-02-28 16:36:13 +01:00
David Kyle 59b50bb18c [ML] Index template registry with simple versioning (elastic/x-pack-elasticsearch#655)
[ML] * Add MachineLearningTemplateRegistry class

[ML] * Add blocking method to put templates required by tests

[ML]  * Add version check for templates

[ML] * Review comments

Original commit: elastic/x-pack-elasticsearch@07d315e56d
2017-02-28 15:34:05 +00:00
Martijn van Groningen 6864111acc [ML] Fixed typo
Original commit: elastic/x-pack-elasticsearch@e66e7febe2
2017-02-28 14:56:43 +01:00
Martijn van Groningen b7b24ed877 [ML] Removed norelease: Don't specify an id for certain documents, but let ES generate an id.
This norelease can be removed, because we no longer create indices automatically when missing, but use index templates.

Original commit: elastic/x-pack-elasticsearch@b6880ce7b7
2017-02-28 14:38:57 +01:00
Martijn van Groningen 759f0b1281 [ML] Close job api should use provided client instead of transport actions directly
Original commit: elastic/x-pack-elasticsearch@cb92f24ee1
2017-02-28 13:44:21 +01:00
David Kyle fc35b6022f Fix checkstyle long lines
Original commit: elastic/x-pack-elasticsearch@7fa810292f
2017-02-28 09:42:36 +00:00
Martijn van Groningen 93a2a567cb [ML] Cut open job and start datafeed apis to use persistent task's assignment explanation.
Also included assignment explanation to both job and datafeed stats apis and
included executor node to datafeed stats api

Original commit: elastic/x-pack-elasticsearch@783bc77ef6
2017-02-28 10:38:51 +01:00
David Kyle 69880373fc Remove the ml_messages.properties file (elastic/x-pack-elasticsearch#657)
String constants are no in Messages

Original commit: elastic/x-pack-elasticsearch@ab8af83e10
2017-02-28 09:33:47 +00:00
Dimitris Athanasiou 2d27e8ad81 [ML] Warn when lookback only datafeed retrieved no data (elastic/x-pack-elasticsearch#651)
When the datafeed indexes are concrete indexes, the search
will throw IndexNotFoundException, thus a notification is being
generated. However, when the indexes are patterns, the search
does not fail.

This commit adds a warning when a lookback-only datafeed
retrieves no data as it is certain that the user misconfigured
something and it's useful to make him aware of it.

Original commit: elastic/x-pack-elasticsearch@e3490ea978
2017-02-27 14:18:07 +00:00
javanna adc9808eff Convert watcher search source to json
If any format other than json is sent, it should get converted to json, as watcher works under the assumption that the script is a template, hence json.

Original commit: elastic/x-pack-elasticsearch@554bcd306b
2017-02-27 12:28:04 +01:00
javanna 39ccc67a7b disable content-type randomization for some ml tests
post_data and get_model seem to support json only

Original commit: elastic/x-pack-elasticsearch@683f998228
2017-02-27 12:28:04 +01:00
javanna 33ccc3bd6c adapt to ObjectPath changes to support binary formats
Original commit: elastic/x-pack-elasticsearch@11782418e9
2017-02-27 12:28:04 +01:00
Martijn van Groningen 212098fc7c [TEST] Attempt to stabilize distributed ml tests.
Relates to elastic/x-pack-elasticsearch#653

Original commit: elastic/x-pack-elasticsearch@e4919ab2b4
2017-02-27 11:11:19 +01:00
David Roberts e2f5715191 [TEST] Fix one last test that was missed in elastic/x-pack-elasticsearch#649
Original commit: elastic/x-pack-elasticsearch@1039197afd
2017-02-27 09:55:49 +00:00
David Roberts 2ca1ddf017 [ML] Tell native controller process to stop during graceful shutdown (elastic/x-pack-elasticsearch#644)
Fixes elastic/prelert-legacy#803

On Windows the JVM hangs during shutdown if the native controller process
is still running. A thread dump suggests this is because of a finalizer being
blocked. We can avoid the problem by explicitly telling the controller to exit
when the node is stopped. Although the hang is Windows-specific, it doesn't
hurt to do this on *nix too.

Original commit: elastic/x-pack-elasticsearch@6fa1c2b966
2017-02-27 08:52:03 +00:00
Colin Goodheart-Smithe 4a81c85f71 Reapplying: Adds cast hint for eclipse
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners

This fix was mistakenly removed in another commit

Original commit: elastic/x-pack-elasticsearch@6305456d5a
2017-02-27 08:43:27 +00:00
Dimitris Athanasiou 380f6d9bff [ML] Add datafeed preview end-point (elastic/x-pack-elasticsearch#645)
Adds a preview end-point: `/_xpack/ml/datafeeds/{datafeed_id}/_preview`

The endpoint returns an array with the JSON records that will reach
writer to the c++ process. Thus, the preview can be used to verify
that a datafeed is configured correctly with regard to itself and
its corresponding job.

Original commit: elastic/x-pack-elasticsearch@1e0e9c906d
2017-02-25 10:17:06 +00:00
Dimitris Athanasiou d49930040e [TEST] Disable autodetect_process in tests not needing it (elastic/x-pack-elasticsearch#649)
Original commit: elastic/x-pack-elasticsearch@38897c0040
2017-02-24 18:57:33 +00:00
David Kyle 08b7cb6bca [ML] Refactor Auditor (elastic/x-pack-elasticsearch#631)
* Refactor Auditor
* Inject Auditor


Original commit: elastic/x-pack-elasticsearch@d9efe5fcef
2017-02-24 15:56:38 +00:00
Martijn van Groningen a189e1b759 [TEST] Added more failure scenarios
Original commit: elastic/x-pack-elasticsearch@a27134323e
2017-02-24 16:35:28 +01:00
David Kyle 8a3706f395 [ML] Influencers come after analysis fields (elastic/x-pack-elasticsearch#646)
Original commit: elastic/x-pack-elasticsearch@f37835b140
2017-02-24 15:12:35 +00:00
Martijn van Groningen 8ecdf3b288 [TEST] Attempt to reduce `MockPageCacheRecycler` assertion failures by waiting until there are no active recoveries.
Original commit: elastic/x-pack-elasticsearch@1b142f29bc
2017-02-24 15:23:02 +01:00
Martijn van Groningen 2c5e32f3d3 [ML] Removed workaround that task status named writable needed to have the same name as action name.
Original commit: elastic/x-pack-elasticsearch@99d9d80073
2017-02-24 14:32:53 +01:00
Martijn van Groningen cd4e543479 [ML] Tweaked starting datafeed logic, so that it will not start when job hasn't started.
[TEST] Unmuted ml full cluster restart test.

Original commit: elastic/x-pack-elasticsearch@c8cd9870c9
2017-02-24 13:47:44 +01:00
David Kyle f67d8c3746 [ML] Fix intermittent test failure (occasionally deleted an index we don’t care about)
Original commit: elastic/x-pack-elasticsearch@56c7bdb5b9
2017-02-24 12:20:47 +00:00
David Kyle a09e8a260a [ML] Use Index Templates (elastic/x-pack-elasticsearch#622)
* [ML] Index template for notifications, meta and state indices

* Add ignore_unavailable option to searches so missing indices don’t throw errors

* Test for job existence before returning results

* Template mapping for results indices

* Fix tests

* Reinstate merge conflict dropped code

* Fix and tidy up JobProviderTests

Tests had badly named & duplicated tests

* Don’t check if the job indices already exist when allocating

Indices are created on demand from templates

* No need to create indices in initialisation service

Using templates now

* Remove unused code and add comment

Original commit: elastic/x-pack-elasticsearch@1ba115d4fc
2017-02-24 10:20:07 +00:00
Colin Goodheart-Smithe 7f6edbdd4b Adds cast hint for eclipse
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners

Original commit: elastic/x-pack-elasticsearch@a7aff45571
2017-02-24 08:38:33 +00:00
Igor Motov 615dd0f240 Request and Status in Persistent Tasks should be serialized using their writable names
Refactors xcontent serialization of Request and Status to use their writable names instead of action name. That simplifies the parsing logic, allows reuse of the same status object for multiple actions and is consistent with how named objects in xcontent are used.

Original commit: elastic/x-pack-elasticsearch@3d059ea952
2017-02-23 18:55:38 -05:00
Martijn van Groningen 9f48a1f677 [ML] Added the node the job is running on to the response of the job stats api.
Original commit: elastic/x-pack-elasticsearch@d92149ca59
2017-02-23 20:48:40 +01:00
Jay Modi 405230c308 Do not add null listeners to the DnRoleMapper (elastic/x-pack-elasticsearch#636)
The DnRoleMapper was changed when moving to asynchronous authentication, which introduced a regression that leads to a
NPE being thrown on file refresh. The cause is the addition of a null listener from the LdapRealm upon construction.

This change removes the listener constructor argument as it was always null or a empty runnable and adds a check in the
addListener method to ensure the listener is not null. Finally, the DnRoleMapperTests had a bug where the temporary
filenames did not have the correct suffix and tests were not actually testing what was intended.

relates elastic/x-pack-elasticsearch#608

Original commit: elastic/x-pack-elasticsearch@f47f258590
2017-02-23 11:16:14 -07:00
Ali Beyad 4a001706e7 Simplifies security index state changes and template/mapping updates (elastic/x-pack-elasticsearch#551)
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.

This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.

This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.

Original commit: elastic/x-pack-elasticsearch@b31d144597
2017-02-23 13:05:39 -05:00
jaymode e61f87d5ff Test: do not use the same name twice
In rare cases, the CertificateToolTests#testPromptingForInstanceInformation could try to use the same name multiple
times. This causes the test to fail with a message for an unknown prompt. This commit ensures no duplicates are added.

relates elastic/x-pack-elasticsearch#597

Original commit: elastic/x-pack-elasticsearch@ab8bb7ad50
2017-02-23 12:34:06 -05:00
Igor Motov db88cc458d Add persistent task assignment explanations.
This commit allows persistent actions to indicate why a task was or wasn't assigned to a certain node.

Original commit: elastic/x-pack-elasticsearch@cdacf9b10f
2017-02-23 11:22:08 -05:00
Martijn van Groningen d87926ab86 [ML] Never stop the datafeeder when no data can be found
Changed `ProblemTracker#updateEmptyDataCount(boolean)` into `ProblemTracker#reportEmptyDataCount()`.
and added `ProblemTracker#reportNoneEmptyCount()` to reset the empty count counter.

Original commit: elastic/x-pack-elasticsearch@c1a44d6fd3
2017-02-23 16:32:08 +01:00
Martijn van Groningen e7d56e92f8 [Ml] Rename `useNativeProcess` setting to `xpack.ml.autodetect_process` and un-deprecate it.
Original commit: elastic/x-pack-elasticsearch@eb2143e6f0
2017-02-23 15:29:44 +01:00
Dimitris Athanasiou 412d5e35b1 [ML] model_debug_config: add enabled and remove bounds_percentile (elastic/x-pack-elasticsearch#627)
This allows it to be disabled/enabled via job updates.
It also simplifies it by removing bounds_percentile as it currently
rarely makes sense to set the bounds to another value than 95.0.

Original commit: elastic/x-pack-elasticsearch@c27fce2d86
2017-02-23 14:11:23 +00:00
David Roberts 33b10970ee [ML] Temporary workaround to make some non-ml integration tests work (elastic/x-pack-elasticsearch#630)
This will be replaced with a better solution when elastic/x-pack-elasticsearch#566 is implemented

Original commit: elastic/x-pack-elasticsearch@bf0d9e0176
2017-02-23 11:13:58 +00:00
Dimitris Athanasiou d3d73ca095 [ML] Remove data that expired their respective retention period (elastic/x-pack-elasticsearch#610)
This PR introduces a service component that is scheduled to run
once per day, around 30 minutes past midnight, local time.
Listeners can be subscribed to the service in order to get notified
when the next service trigger occurs so that they can execute
management tasks.

Currently, there are 2 such listeners: one is performing removal
of results and the other of model snapshots that have expired
their retention period.

Finally, note that the service is only installed on the master
node to ensure there is only one running on the cluster.

Original commit: elastic/x-pack-elasticsearch@c8edb6016a
2017-02-23 11:06:14 +00:00
Martijn van Groningen a03e9d36c2 [TEST] mute test
Original commit: elastic/x-pack-elasticsearch@3989e5a074
2017-02-23 11:38:44 +01:00
Martijn van Groningen b40d5dde9b [ML] Before selecting an executor node for running job task, make sure that all required indices exist and are accessible.
Original commit: elastic/x-pack-elasticsearch@5459a0d5f2
2017-02-23 10:55:16 +01:00
Alexander Reelsen 43a09a2fa7 Watcher: Add license files (elastic/x-pack-elasticsearch#619)
Add license, notice and SHA sums for watcher compile dependencies
(excluding test deps).

Relates elastic/x-pack-elasticsearch#607

Original commit: elastic/x-pack-elasticsearch@06ce510436
2017-02-23 10:24:16 +01:00
Martijn van Groningen 2eba41cfac [TEST] Fix mocking bug
Original commit: elastic/x-pack-elasticsearch@61268c1180
2017-02-23 09:59:45 +01:00
Martijn van Groningen 6195000b8a [TEST] Change ml cleanup workaround.
Original commit: elastic/x-pack-elasticsearch@793d6cfb78
2017-02-23 09:08:10 +01:00