Commit Graph

1242 Commits

Author SHA1 Message Date
Luca Cavanna 70a4a69e39 Fix indentation in some yaml tests (elastic/x-pack-elasticsearch#686)
The yaml test runner now throws error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which was previously ignored. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.

Original commit: elastic/x-pack-elasticsearch@920201207c
2017-03-02 12:43:43 +01:00
Martijn van Groningen 0f5bfca69a [TEST] wait for yellow, to give ml indices time to get ready, open job api fails otherwise
Original commit: elastic/x-pack-elasticsearch@b1002cd289
2017-03-02 09:50:45 +01:00
Martijn van Groningen b5db0d9f44 [TEST] Change `DatafeedJobsIT` test to use the external cluster as it doesn't start/stop nodes.
Only distributed ml tests, only enable ml as security, watcher, monitoring etc. only add noise to the tests.
Disable mock big arrays for distributed ml tests, as nodes starting/stopping in tests somehow leaves allocations around,
which only is a test problem.

Original commit: elastic/x-pack-elasticsearch@5ff1e69036
2017-03-01 23:09:57 +01:00
Zachary Tong eed0e41a29 [ML] Share job result indices by default (elastic/x-pack-elasticsearch#638)
This moves the index structure to using a single, shared index
(.ml-anomalies-shared).  Custom indices can still be used by manually
setting `results_index`.

An alias is always created which points from `.ml-anomalies-<jobid>`
to `.ml-anomalies-shared`.

User defined indices are prepended with "custom-"

Index helper functions have been renamed to make this clear.  Furthermore,
accessing an index should always be done either by fetching the
currently configured index/alias from the state, or using the preconfigured
alias.  Because the user can specify a custom physical index, it is
impossible to determine the physical index "by convention" now.
The helpers have been configured to reflect that.

Original commit: elastic/x-pack-elasticsearch@a5368eb230
2017-03-01 13:28:12 -05:00
Ali Beyad b20578b9f6 Enables X-Pack extensions to implement custom roles providers (elastic/x-pack-elasticsearch#603)
This commit adds the ability for x-pack extensions to optionally
provide custom roles providers, which are used to resolve any roles
into role descriptors that are not found in the reserved or native
realms.  This feature enables the ability to define and provide roles
from other sources, without having to pre-define such roles in the security
config files.

relates elastic/x-pack-elasticsearch#77

Original commit: elastic/x-pack-elasticsearch@bbbe7a49bf
2017-03-01 12:20:34 -05:00
Jay Modi 02579c7acc IteratingActionListener should store context before calling consumers (elastic/x-pack-elasticsearch#675)
As part of authentication, we use a iterating action listener to perform asynchronous authentication against the realm
chain. When this listener is called with a response or a failure, it could be called from a thread that is not owned by
the Elasticsearch threadpool such as a LDAPConnectionReader thread. When this happens, we need to ensure that the
ThreadContext is not left with items in it otherwise we leave behind things like Authentication and hit obscure errors.

This commit stores the context when the listener calls the consumer or onResponse/onFailure is invoked, which prevents
us from polluting a external thread's ThreadContext.

Original commit: elastic/x-pack-elasticsearch@0f50fb6c10
2017-03-01 10:40:42 -05:00
Martijn van Groningen 61ca6d435f [ML] Enforce the stop datafeed aoi to wait until any operation that the datafeed is doing has completed.
Original commit: elastic/x-pack-elasticsearch@e8974191a2
2017-03-01 15:14:41 +01:00
Alexander Reelsen e492b17c10 Watcher: Ensure all json builders use try-with-resources (elastic/x-pack-elasticsearch#650)
Some json builders in the codebase were not closed. even
though this is not needed for the BytesStreamOutput being used,
there is more closing logic in the jackson classes, which we
should not rely on, that those never change or are ok to not
close.

Original commit: elastic/x-pack-elasticsearch@05a43d80ff
2017-03-01 13:45:10 +01:00
David Kyle 4edfbe664f [ML] Delete unused AuditActivity class (elastic/x-pack-elasticsearch#672)
Original commit: elastic/x-pack-elasticsearch@185ceab4b4
2017-03-01 10:42:27 +00:00
David Roberts 094be3c28b [ML] Put C++ process/PID and file@line in square brackets (elastic/x-pack-elasticsearch#671)
This is the Elastic convention, and also makes it clearer where the
actual log message from the C++ starts

Original commit: elastic/x-pack-elasticsearch@dc9aeefb08
2017-03-01 09:33:30 +00:00
Martijn van Groningen ac93195407 [TEST] Don't install xpack plugin as the test doesn't rely on it initializes the ml classes it tests on its own.
Original commit: elastic/x-pack-elasticsearch@e505efcddb
2017-03-01 09:32:34 +01:00
Igor Motov 377c1ec2b4 Simplify names of PersistentTasks-related classes
PersistentTask -> NodePersistentTask
PersistentTasksInProgress -> PersistentTasks
PersistentTaskInProgress -> PersistentTask

Original commit: elastic/x-pack-elasticsearch@0947dbca56
2017-02-28 16:37:09 -07:00
Martijn van Groningen 9d6c344520 [ML] Disable ml in case a node is a tribe node and
in case of tribe node client do not even bother to check for native process version, because it isn't running.

Original commit: elastic/x-pack-elasticsearch@e6fdd5e82a
2017-02-28 21:31:09 +01:00
Martijn van Groningen 6783f823a8 [ML] Change stop datafeed api delegate to node hosting datafeed task and execute cancel locally,
instead of only removing the persistent task from cluster state.

Original commit: elastic/x-pack-elasticsearch@3974b20827
2017-02-28 16:36:13 +01:00
David Kyle 59b50bb18c [ML] Index template registry with simple versioning (elastic/x-pack-elasticsearch#655)
[ML] * Add MachineLearningTemplateRegistry class

[ML] * Add blocking method to put templates required by tests

[ML]  * Add version check for templates

[ML] * Review comments

Original commit: elastic/x-pack-elasticsearch@07d315e56d
2017-02-28 15:34:05 +00:00
Martijn van Groningen 6864111acc [ML] Fixed typo
Original commit: elastic/x-pack-elasticsearch@e66e7febe2
2017-02-28 14:56:43 +01:00
Martijn van Groningen b7b24ed877 [ML] Removed norelease: Don't specify an id for certain documents, but let ES generate an id.
This norelease can be removed, because we no longer create indices automatically when missing, but use index templates.

Original commit: elastic/x-pack-elasticsearch@b6880ce7b7
2017-02-28 14:38:57 +01:00
Martijn van Groningen 759f0b1281 [ML] Close job api should use provided client instead of transport actions directly
Original commit: elastic/x-pack-elasticsearch@cb92f24ee1
2017-02-28 13:44:21 +01:00
David Kyle fc35b6022f Fix checkstyle long lines
Original commit: elastic/x-pack-elasticsearch@7fa810292f
2017-02-28 09:42:36 +00:00
Martijn van Groningen 93a2a567cb [ML] Cut open job and start datafeed apis to use persistent task's assignment explanation.
Also included assignment explanation to both job and datafeed stats apis and
included executor node to datafeed stats api

Original commit: elastic/x-pack-elasticsearch@783bc77ef6
2017-02-28 10:38:51 +01:00
David Kyle 69880373fc Remove the ml_messages.properties file (elastic/x-pack-elasticsearch#657)
String constants are no in Messages

Original commit: elastic/x-pack-elasticsearch@ab8af83e10
2017-02-28 09:33:47 +00:00
Dimitris Athanasiou 2d27e8ad81 [ML] Warn when lookback only datafeed retrieved no data (elastic/x-pack-elasticsearch#651)
When the datafeed indexes are concrete indexes, the search
will throw IndexNotFoundException, thus a notification is being
generated. However, when the indexes are patterns, the search
does not fail.

This commit adds a warning when a lookback-only datafeed
retrieves no data as it is certain that the user misconfigured
something and it's useful to make him aware of it.

Original commit: elastic/x-pack-elasticsearch@e3490ea978
2017-02-27 14:18:07 +00:00
javanna adc9808eff Convert watcher search source to json
If any format other than json is sent, it should get converted to json, as watcher works under the assumption that the script is a template, hence json.

Original commit: elastic/x-pack-elasticsearch@554bcd306b
2017-02-27 12:28:04 +01:00
javanna 39ccc67a7b disable content-type randomization for some ml tests
post_data and get_model seem to support json only

Original commit: elastic/x-pack-elasticsearch@683f998228
2017-02-27 12:28:04 +01:00
javanna 33ccc3bd6c adapt to ObjectPath changes to support binary formats
Original commit: elastic/x-pack-elasticsearch@11782418e9
2017-02-27 12:28:04 +01:00
Martijn van Groningen 212098fc7c [TEST] Attempt to stabilize distributed ml tests.
Relates to elastic/x-pack-elasticsearch#653

Original commit: elastic/x-pack-elasticsearch@e4919ab2b4
2017-02-27 11:11:19 +01:00
David Roberts 2ca1ddf017 [ML] Tell native controller process to stop during graceful shutdown (elastic/x-pack-elasticsearch#644)
Fixes elastic/prelert-legacy#803

On Windows the JVM hangs during shutdown if the native controller process
is still running. A thread dump suggests this is because of a finalizer being
blocked. We can avoid the problem by explicitly telling the controller to exit
when the node is stopped. Although the hang is Windows-specific, it doesn't
hurt to do this on *nix too.

Original commit: elastic/x-pack-elasticsearch@6fa1c2b966
2017-02-27 08:52:03 +00:00
Colin Goodheart-Smithe 4a81c85f71 Reapplying: Adds cast hint for eclipse
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners

This fix was mistakenly removed in another commit

Original commit: elastic/x-pack-elasticsearch@6305456d5a
2017-02-27 08:43:27 +00:00
Dimitris Athanasiou 380f6d9bff [ML] Add datafeed preview end-point (elastic/x-pack-elasticsearch#645)
Adds a preview end-point: `/_xpack/ml/datafeeds/{datafeed_id}/_preview`

The endpoint returns an array with the JSON records that will reach
writer to the c++ process. Thus, the preview can be used to verify
that a datafeed is configured correctly with regard to itself and
its corresponding job.

Original commit: elastic/x-pack-elasticsearch@1e0e9c906d
2017-02-25 10:17:06 +00:00
Dimitris Athanasiou d49930040e [TEST] Disable autodetect_process in tests not needing it (elastic/x-pack-elasticsearch#649)
Original commit: elastic/x-pack-elasticsearch@38897c0040
2017-02-24 18:57:33 +00:00
David Kyle 08b7cb6bca [ML] Refactor Auditor (elastic/x-pack-elasticsearch#631)
* Refactor Auditor
* Inject Auditor


Original commit: elastic/x-pack-elasticsearch@d9efe5fcef
2017-02-24 15:56:38 +00:00
Martijn van Groningen a189e1b759 [TEST] Added more failure scenarios
Original commit: elastic/x-pack-elasticsearch@a27134323e
2017-02-24 16:35:28 +01:00
David Kyle 8a3706f395 [ML] Influencers come after analysis fields (elastic/x-pack-elasticsearch#646)
Original commit: elastic/x-pack-elasticsearch@f37835b140
2017-02-24 15:12:35 +00:00
Martijn van Groningen 8ecdf3b288 [TEST] Attempt to reduce `MockPageCacheRecycler` assertion failures by waiting until there are no active recoveries.
Original commit: elastic/x-pack-elasticsearch@1b142f29bc
2017-02-24 15:23:02 +01:00
Martijn van Groningen 2c5e32f3d3 [ML] Removed workaround that task status named writable needed to have the same name as action name.
Original commit: elastic/x-pack-elasticsearch@99d9d80073
2017-02-24 14:32:53 +01:00
Martijn van Groningen cd4e543479 [ML] Tweaked starting datafeed logic, so that it will not start when job hasn't started.
[TEST] Unmuted ml full cluster restart test.

Original commit: elastic/x-pack-elasticsearch@c8cd9870c9
2017-02-24 13:47:44 +01:00
David Kyle f67d8c3746 [ML] Fix intermittent test failure (occasionally deleted an index we don’t care about)
Original commit: elastic/x-pack-elasticsearch@56c7bdb5b9
2017-02-24 12:20:47 +00:00
David Kyle a09e8a260a [ML] Use Index Templates (elastic/x-pack-elasticsearch#622)
* [ML] Index template for notifications, meta and state indices

* Add ignore_unavailable option to searches so missing indices don’t throw errors

* Test for job existence before returning results

* Template mapping for results indices

* Fix tests

* Reinstate merge conflict dropped code

* Fix and tidy up JobProviderTests

Tests had badly named & duplicated tests

* Don’t check if the job indices already exist when allocating

Indices are created on demand from templates

* No need to create indices in initialisation service

Using templates now

* Remove unused code and add comment

Original commit: elastic/x-pack-elasticsearch@1ba115d4fc
2017-02-24 10:20:07 +00:00
Colin Goodheart-Smithe 7f6edbdd4b Adds cast hint for eclipse
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners

Original commit: elastic/x-pack-elasticsearch@a7aff45571
2017-02-24 08:38:33 +00:00
Igor Motov 615dd0f240 Request and Status in Persistent Tasks should be serialized using their writable names
Refactors xcontent serialization of Request and Status to use their writable names instead of action name. That simplifies the parsing logic, allows reuse of the same status object for multiple actions and is consistent with how named objects in xcontent are used.

Original commit: elastic/x-pack-elasticsearch@3d059ea952
2017-02-23 18:55:38 -05:00
Martijn van Groningen 9f48a1f677 [ML] Added the node the job is running on to the response of the job stats api.
Original commit: elastic/x-pack-elasticsearch@d92149ca59
2017-02-23 20:48:40 +01:00
Jay Modi 405230c308 Do not add null listeners to the DnRoleMapper (elastic/x-pack-elasticsearch#636)
The DnRoleMapper was changed when moving to asynchronous authentication, which introduced a regression that leads to a
NPE being thrown on file refresh. The cause is the addition of a null listener from the LdapRealm upon construction.

This change removes the listener constructor argument as it was always null or a empty runnable and adds a check in the
addListener method to ensure the listener is not null. Finally, the DnRoleMapperTests had a bug where the temporary
filenames did not have the correct suffix and tests were not actually testing what was intended.

relates elastic/x-pack-elasticsearch#608

Original commit: elastic/x-pack-elasticsearch@f47f258590
2017-02-23 11:16:14 -07:00
Ali Beyad 4a001706e7 Simplifies security index state changes and template/mapping updates (elastic/x-pack-elasticsearch#551)
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.

This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.

This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.

Original commit: elastic/x-pack-elasticsearch@b31d144597
2017-02-23 13:05:39 -05:00
jaymode e61f87d5ff Test: do not use the same name twice
In rare cases, the CertificateToolTests#testPromptingForInstanceInformation could try to use the same name multiple
times. This causes the test to fail with a message for an unknown prompt. This commit ensures no duplicates are added.

relates elastic/x-pack-elasticsearch#597

Original commit: elastic/x-pack-elasticsearch@ab8bb7ad50
2017-02-23 12:34:06 -05:00
Igor Motov db88cc458d Add persistent task assignment explanations.
This commit allows persistent actions to indicate why a task was or wasn't assigned to a certain node.

Original commit: elastic/x-pack-elasticsearch@cdacf9b10f
2017-02-23 11:22:08 -05:00
Martijn van Groningen d87926ab86 [ML] Never stop the datafeeder when no data can be found
Changed `ProblemTracker#updateEmptyDataCount(boolean)` into `ProblemTracker#reportEmptyDataCount()`.
and added `ProblemTracker#reportNoneEmptyCount()` to reset the empty count counter.

Original commit: elastic/x-pack-elasticsearch@c1a44d6fd3
2017-02-23 16:32:08 +01:00
Martijn van Groningen e7d56e92f8 [Ml] Rename `useNativeProcess` setting to `xpack.ml.autodetect_process` and un-deprecate it.
Original commit: elastic/x-pack-elasticsearch@eb2143e6f0
2017-02-23 15:29:44 +01:00
Dimitris Athanasiou 412d5e35b1 [ML] model_debug_config: add enabled and remove bounds_percentile (elastic/x-pack-elasticsearch#627)
This allows it to be disabled/enabled via job updates.
It also simplifies it by removing bounds_percentile as it currently
rarely makes sense to set the bounds to another value than 95.0.

Original commit: elastic/x-pack-elasticsearch@c27fce2d86
2017-02-23 14:11:23 +00:00
David Roberts 33b10970ee [ML] Temporary workaround to make some non-ml integration tests work (elastic/x-pack-elasticsearch#630)
This will be replaced with a better solution when elastic/x-pack-elasticsearch#566 is implemented

Original commit: elastic/x-pack-elasticsearch@bf0d9e0176
2017-02-23 11:13:58 +00:00
Dimitris Athanasiou d3d73ca095 [ML] Remove data that expired their respective retention period (elastic/x-pack-elasticsearch#610)
This PR introduces a service component that is scheduled to run
once per day, around 30 minutes past midnight, local time.
Listeners can be subscribed to the service in order to get notified
when the next service trigger occurs so that they can execute
management tasks.

Currently, there are 2 such listeners: one is performing removal
of results and the other of model snapshots that have expired
their retention period.

Finally, note that the service is only installed on the master
node to ensure there is only one running on the cluster.

Original commit: elastic/x-pack-elasticsearch@c8edb6016a
2017-02-23 11:06:14 +00:00
Martijn van Groningen a03e9d36c2 [TEST] mute test
Original commit: elastic/x-pack-elasticsearch@3989e5a074
2017-02-23 11:38:44 +01:00
Martijn van Groningen b40d5dde9b [ML] Before selecting an executor node for running job task, make sure that all required indices exist and are accessible.
Original commit: elastic/x-pack-elasticsearch@5459a0d5f2
2017-02-23 10:55:16 +01:00
Alexander Reelsen 43a09a2fa7 Watcher: Add license files (elastic/x-pack-elasticsearch#619)
Add license, notice and SHA sums for watcher compile dependencies
(excluding test deps).

Relates elastic/x-pack-elasticsearch#607

Original commit: elastic/x-pack-elasticsearch@06ce510436
2017-02-23 10:24:16 +01:00
Martijn van Groningen 2eba41cfac [TEST] Fix mocking bug
Original commit: elastic/x-pack-elasticsearch@61268c1180
2017-02-23 09:59:45 +01:00
Martijn van Groningen 6195000b8a [TEST] Change ml cleanup workaround.
Original commit: elastic/x-pack-elasticsearch@793d6cfb78
2017-02-23 09:08:10 +01:00
Tim Vernum 9f6285f362 [TEST] Waiting longer for native realm migration
Let LicensingTests.testNativeRealmMigratorWorksUnderBasicLicense wait for 3s (instead of 1s) as it is occasionally failing.
Also added logging so that if/when it does fail we can see how long we were waiting and what was taking place.

Original commit: elastic/x-pack-elasticsearch@22882ec412
2017-02-23 14:15:17 +11:00
Tim Vernum 672819f038 LICENSE files for security (elastic/x-pack-elasticsearch#614)
* License information for unboundid-ldapsdk
   From: https://docs.ldap.com/ldap-sdk/docs/LICENSE-UnboundID-LDAPSDK.txt

* License information for Bouncy Castle
   From:  java -cp ~/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.55/bcprov-jdk15on-1.55.jar org.bouncycastle.LICENSE


Original commit: elastic/x-pack-elasticsearch@b32d5aade3
2017-02-23 10:43:34 +11:00
Ryan Ernst bca7d87ac4 Rename threadpool time method
See elastic/elasticsearch#23175

Original commit: elastic/x-pack-elasticsearch@0e185c0a44
2017-02-22 15:11:42 -08:00
Dimitris Athanasiou eda2f15b44 [ML] Remove deprecated use of TransportDeleteAction (elastic/x-pack-elasticsearch#628)
Original commit: elastic/x-pack-elasticsearch@b5fdb32a0c
2017-02-22 22:14:32 +00:00
Martijn van Groningen 783a2cf01b [TEST] Added more test logging
Original commit: elastic/x-pack-elasticsearch@d68a1e3510
2017-02-22 21:47:01 +01:00
Ryan Ernst 2cd3979654 Merge pull request elastic/x-pack-elasticsearch#617 from rjernst/integ_test_order
Build: Convert integ test dsl to new split cluster/runner dsl

Original commit: elastic/x-pack-elasticsearch@454edd48d1
2017-02-22 12:43:25 -08:00
Martijn van Groningen 124fe0ad04 [ML] Fix line length
Original commit: elastic/x-pack-elasticsearch@2fba2434b8
2017-02-22 21:37:25 +01:00
polyfractal 4041545cfa [ML] Put back check for inconsistent job id
Lost during the refactor, this checks to make sure an ID in both the URL and the
body match.

Original commit: elastic/x-pack-elasticsearch@ef96275c99
2017-02-22 15:21:03 -05:00
Martijn van Groningen 9da18a14c5 [ML] Set createTime if not specified in validate api too
Original commit: elastic/x-pack-elasticsearch@477fb0fc52
2017-02-22 21:03:55 +01:00
polyfractal f856a79ef3 Fix method signature in test
Original commit: elastic/x-pack-elasticsearch@4340952f77
2017-02-22 14:08:19 -05:00
Zachary Tong 41811efcf1 [ML] Job's CreateTime must be non-null (elastic/x-pack-elasticsearch#616)
Also simplifies Job.Builder to only have one build() method.

Original commit: elastic/x-pack-elasticsearch@50e4b4c210
2017-02-22 13:45:13 -05:00
Martijn van Groningen 9a5b410110 [TEST] Add test that verifies that machine learning works correctly after full cluster restart.
Added missing named xcontent entries, tweaked start datafeed validation logic and fixed status serialization issue that were uncovered by this test.

Original commit: elastic/x-pack-elasticsearch@8b3fb012df
2017-02-22 17:25:08 +01:00
Dimitrios Athanasiou 4bb57137a9 fix unit test
Original commit: elastic/x-pack-elasticsearch@af9276f02f
2017-02-22 14:42:39 +00:00
Dimitrios Athanasiou a6c62f0717 [ML] Search model state docs using correct id
Original commit: elastic/x-pack-elasticsearch@70400101ad
2017-02-22 13:54:18 +00:00
David Kyle f06b4fbbaf [ML] remove duplicate code
Original commit: elastic/x-pack-elasticsearch@340480477c
2017-02-22 13:52:38 +00:00
Dimitrios Athanasiou b233c4729d [ML] Correctly fetch persistent tasks in JobUpdateAction
Original commit: elastic/x-pack-elasticsearch@440bfdd06e
2017-02-22 11:26:31 +00:00
David Roberts 614aeeeb53 [ML] Add back SuperCSV license/notice (elastic/x-pack-elasticsearch#618)
These were lost during the migration from prelert-legacy to x-pack

Original commit: elastic/x-pack-elasticsearch@da338d4a44
2017-02-22 10:49:26 +00:00
David Kyle f82634b6b1 [ML] Use ExceptionsHelper.convertToElastic to wrap exceptions in TransportJobTaskAction (elastic/x-pack-elasticsearch#577)
* Wrap exceptions if they are not ElasticsearchExceptions in TransportJobTaskAction

* Review comments

Original commit: elastic/x-pack-elasticsearch@07e900ffaa
2017-02-22 10:03:22 +00:00
Ryan Ernst 8527bc2415 Build: Convert integ test dsl to new split cluster/runner dsl
This is the xpack side of elastic/elasticsearch#23304

Original commit: elastic/x-pack-elasticsearch@8eddd7fb0d
2017-02-22 00:56:52 -08:00
polyfractal 45ecebac20 [Watcher] Fix tests due to upstream method signature change
Caused by elastic/elasticsearch#23288

Original commit: elastic/x-pack-elasticsearch@6f041708f9
2017-02-21 13:48:53 -05:00
Martijn van Groningen 9100fa6333 [ML] Rename `xpack.ml.allocation_enabled` boolean setting to `node.ml`.
Remove `node.attr.xpack.ml.allocation_enabled` node attribute and instead use `node.attr.max_running_jobs` attribute to indicate a node is a ml node.
(in addition to indicating how many jobs can hosted on a node)

Original commit: elastic/x-pack-elasticsearch@7979bc55b4
2017-02-21 14:10:50 +01:00
Dimitris Athanasiou 165c0d0e4f [ML] Move metadata package classes at the top (elastic/x-pack-elasticsearch#605)
Original commit: elastic/x-pack-elasticsearch@579f86e7b9
2017-02-21 13:08:16 +00:00
Dimitrios Athanasiou b9f83da5ac [ML] Include all fields in Job copy constructor
Fields were missing from the job's copy constructor.
This was the cause of intermittent test failures.

Original commit: elastic/x-pack-elasticsearch@3cd705ab3c
2017-02-21 12:49:08 +00:00
Martijn van Groningen 0e44683de6 [TEST] Change default close timeout for close job requests in tests.
Original commit: elastic/x-pack-elasticsearch@240bb0561c
2017-02-21 13:07:25 +01:00
Dimitrios Athanasiou c49a17e7ca [ML] Fix model snapshot YAML tests
Original commit: elastic/x-pack-elasticsearch@e29663da35
2017-02-21 11:42:24 +00:00
Dimitris Athanasiou 6c97ac7691 [ML] Remove model snapshot restore_priority and... (elastic/x-pack-elasticsearch#598)
...correctly update the job's model_snapshot_id whenever
a new model snapshot is persisted. Load the snapshot that
corresponds to the job's model_snapshot_id when a job is
opened.

This PR makes a series of changes that completes the change
of having job's active model snapshot persisted on the job.

Original commit: elastic/x-pack-elasticsearch@c2e23fa1ee
2017-02-20 17:45:28 +00:00
Alexander Reelsen b58b92dc79 Merge pull request elastic/x-pack-elasticsearch#591 from spinscale/1702-rest-hijack-delete-by-query
Delete/Update by query was not disabled for the watcher index
on HTTP.

Test suite was renamed from hijack to `block_direct_index_access` in the watcher tests

relates elastic/x-pack-elasticsearch#586

Original commit: elastic/x-pack-elasticsearch@5984ee3624
2017-02-20 13:24:00 +01:00
David Roberts 782f97a60f Relax regex for getting C++ build so it works for developers with uncommitted changes
Original commit: elastic/x-pack-elasticsearch@a6faea0cc6
2017-02-20 11:25:19 +00:00
Alexander Reelsen 7e8b43bfb1 Merge branch 'master' into 1702-rest-hijack-delete-by-query
Original commit: elastic/x-pack-elasticsearch@c206f84300
2017-02-20 12:06:55 +01:00
David Roberts 5233ab9831 Don't try to get the native code version in the transport client (elastic/x-pack-elasticsearch#600)
The native controller process won't be running

Original commit: elastic/x-pack-elasticsearch@d848e86cd5
2017-02-20 10:56:20 +00:00
Alexander Reelsen d2c705206c renamed test suite
Original commit: elastic/x-pack-elasticsearch@784d5ee68a
2017-02-20 11:43:03 +01:00
Martijn van Groningen 4b1ed5b453 [ML] Add a setting (`xpack.ml.node_concurrent_job_allocations`) to control the maximum concurrent job allocations.
A job allocation is either a job task in OPENING state or a job task that has been assigned to an executor node, but the executor node hasn't had the opportunity to set the job task status to OPENING.

In order to keep track of restarted tasks, `allocationIdOnLastStatusUpdate` field was added to `PersistentTaskInProgress` class.
This will allow persistent task implementors to detect whether the executor node has changed or has been unset since the last status update has occured.

Original commit: elastic/x-pack-elasticsearch@b7b85a8274
2017-02-20 11:22:19 +01:00
David Roberts 0d1181eabb [ML] Show C++ process info in X-Pack's info response (elastic/x-pack-elasticsearch#581)
The /_xpack endpoint now has a "native_code_info" value in the feature
info for ml

Original commit: elastic/x-pack-elasticsearch@6b4b408f4a
2017-02-18 20:14:52 +00:00
Martijn van Groningen 8a6cea0350 [ml] For the data, flush and update apis check the job state on the node running the job task instead of the coordinating node.
[test] Also re-enabled multi node qa tests.

Original commit: elastic/x-pack-elasticsearch@efb24131f0
2017-02-18 19:18:29 +01:00
Martijn van Groningen a74d1329ba [test] changed default close timeout
Original commit: elastic/x-pack-elasticsearch@f07cca869f
2017-02-18 15:40:13 +01:00
Martijn van Groningen 31c88ae1e5 [ml] Set job state to failed when fetching required information for opening a job failed.
Relates to elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@6ad699df86
2017-02-18 13:13:57 +01:00
Jay Modi 68324f5a50 Remove usage of deprecated methods that do not accept a XContentType (elastic/x-pack-elasticsearch#549)
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.

Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.

Original commit: elastic/x-pack-elasticsearch@29d336a008
2017-02-17 14:45:00 -05:00
Martijn van Groningen 22022ea25f [ml] Fix start datafeed validation
relates elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@103fd7f2b4
2017-02-17 20:14:08 +01:00
David Kyle b9a4a2c621 Revert "Don’t wait on job close (elastic/x-pack-elasticsearch#574)"
This reverts commit elastic/x-pack-elasticsearch@99ed0e0dba and fixes the failing tests

Original commit: elastic/x-pack-elasticsearch@403e38316d
2017-02-17 17:05:41 +00:00
Zachary Tong 3fd1c90707 [ML] Move all Job validation to builder and make Job private (elastic/x-pack-elasticsearch#585)
All validation should be done in the builder and the Job object should just be a dumb, immutable blob.

Original commit: elastic/x-pack-elasticsearch@a78e6edbe9
2017-02-17 11:39:41 -05:00
Dimitrios Athanasiou fa9f4e3bca [ML] Remove leftoever debug statement
Original commit: elastic/x-pack-elasticsearch@a3f529b656
2017-02-17 16:16:48 +00:00
David Kyle 45d228dd41 [ML] Synchronise updates to running C++ process (elastic/x-pack-elasticsearch#557)
* Update running process from cluster state update

* Put a big lock on the UpdateProcessAction

* Protect concurrent update of the process with a semaphore

* Review comments

Original commit: elastic/x-pack-elasticsearch@d08b53247a
2017-02-17 16:00:46 +00:00
Martijn van Groningen 75470b4036 [ML] No need to remove the job task manually, when job task is cancelled the persistent task will be removed too.
Original commit: elastic/x-pack-elasticsearch@e0e3947a53
2017-02-17 14:58:00 +01:00
Alexander Reelsen 7d7da4262f Watcher: Hijack REST delete/udpate by query operations
Delete/Update by query was not disabled for the watcher index
on HTTP.

relates elastic/x-pack-elasticsearch#586

Original commit: elastic/x-pack-elasticsearch@b41b375379
2017-02-17 14:32:31 +01:00
Dimitris Athanasiou d970869e00 [ML] Add builder to JobUpdate (elastic/x-pack-elasticsearch#589)
Original commit: elastic/x-pack-elasticsearch@07a69eb407
2017-02-17 11:45:16 +00:00
David Kyle 48ee5e021d Don’t wait on job close (elastic/x-pack-elasticsearch#574)
Original commit: elastic/x-pack-elasticsearch@99ed0e0dba
2017-02-17 09:25:13 +00:00
Dimitris Athanasiou 4f6146cecc [ML] Improve logging on job closing (elastic/x-pack-elasticsearch#587)
Original commit: elastic/x-pack-elasticsearch@58d814fd71
2017-02-17 09:20:23 +00:00
Igor Motov d733aa684d Make persistent task persist full cluster restart
This commit moves persistent tasks from ClusterState.Custom to MetaData.Custom and adds ability for the task to remain in the metadata after completion.

Original commit: elastic/x-pack-elasticsearch@751b40de2a
2017-02-16 19:35:06 -05:00
Zach f4e2830ab1 [ML] Fix MlJobIT#testCreateJobInSharedIndex: "index_name" is now "results_index_name"
relates elastic/x-pack-elasticsearch#584

Original commit: elastic/x-pack-elasticsearch@fbf9fdd9b0
2017-02-16 18:21:13 -05:00
Igor Motov 11a46dd938 [ML] Suppress MlJobIT.testCreateJobInSharedIndexUpdatesMapping
Original commit: elastic/x-pack-elasticsearch@214ff88ac3
2017-02-16 14:24:32 -05:00
Dimitris Athanasiou 1ae8ed495e [ML] Add keyword mapping to audit message to enable aggs (elastic/x-pack-elasticsearch#583)
The messages stored in the .ml-notifications index include a
message field that is mapped to text. It makes sense to keep
that mapping as it can improve search experience in messages.
However, the ML kibana app needs to aggregate over messages,
thus we also need to provide a keyword mapping.

Original commit: elastic/x-pack-elasticsearch@5a5df53d85
2017-02-16 17:53:26 +00:00
Dimitris Athanasiou 6edc051e8d [ML] Close job after a lookback-only datafeed stops (elastic/x-pack-elasticsearch#580)
Original commit: elastic/x-pack-elasticsearch@febf026ba9
2017-02-16 17:48:57 +00:00
David Kyle 15987f49c0 [ML] Update index mapping for jobs in shared indices (elastic/x-pack-elasticsearch#572)
* Update input field mappings if job results index exists

* Add field names to the index mapping in shared indices

* Throw error if creating job will violate the index.mapping.total_fields.limit

* Review comments

Original commit: elastic/x-pack-elasticsearch@d7712d9263
2017-02-16 17:34:47 +00:00
jaymode 6ec6db8544 Test: handle case where multiple state_uuids are found in monitoring shards docs
Original commit: elastic/x-pack-elasticsearch@8d95776081
2017-02-16 11:34:17 -05:00
David Kyle c5506b7440 Consistent defaults for _open job (elastic/x-pack-elasticsearch#578)
Matches OpenJobAction

Original commit: elastic/x-pack-elasticsearch@24c07913a1
2017-02-16 15:41:20 +00:00
David Kyle 545921698c Update in use messages (elastic/x-pack-elasticsearch#576)
[ML] Update in use messages

Original commit: elastic/x-pack-elasticsearch@629961f1ed
2017-02-16 14:44:52 +00:00
jaymode ff96939c5f Authentication should set the proper version on the stream
This commit ensures that the authentication sets the correct version on the stream when it is serialized over the wire
so that there is not a version mismatch between the authentication and the connection it came from.

Original commit: elastic/x-pack-elasticsearch@267d7068f4
2017-02-16 08:44:29 -05:00
Martijn van Groningen 3572cff0a8 [TEST] slightly changed workaround cleanup for ml integ tests.
Original commit: elastic/x-pack-elasticsearch@3eba1e76ab
2017-02-16 14:28:06 +01:00
Martijn van Groningen 55b2111bdf [TEST] wait number of nodes instead of green
Original commit: elastic/x-pack-elasticsearch@d5e47d4e28
2017-02-16 13:54:30 +01:00
Tim Vernum de10d26bfc [Security] Treat LDAP exceptions as auth failures (elastic/x-pack-elasticsearch#565)
Any failure to communicate with an LDAP server should be treated as if authentication failed rather than a fatal "stop the world" error.

Because the AD realm validates the user's password by attempting to bind to the directory with that username/password combination, connection failures are to be expected and must be handled gracefully because the AD realm may not be the last realm in the order-chain, and any exception would prevent later realms from being tested.

Closes: elastic/x-pack-elasticsearch#564

Original commit: elastic/x-pack-elasticsearch@5282290b00
2017-02-16 22:57:12 +11:00
David Kyle e77a71827d Fix integ test after field rename
Original commit: elastic/x-pack-elasticsearch@4ebb93fb44
2017-02-16 11:19:15 +00:00
David Kyle 26df31a3d0 Rename Job.index_name -> Job.results_index_name (elastic/x-pack-elasticsearch#570)
Original commit: elastic/x-pack-elasticsearch@50fc6515e7
2017-02-16 10:26:38 +00:00
Martijn van Groningen d03c6befd5 [ML] Don't set node attributes when ml is disabled.
Original commit: elastic/x-pack-elasticsearch@d450750f41
2017-02-16 10:52:30 +01:00
Colin Goodheart-Smithe 19fc532961 Restricts certain ml endpoints if license forbids ml (elastic/x-pack-elasticsearch#568)
This change adds checks to the following machine learning actions to check that machine learning is permitted by the license before executing the request and throw and exception if the license forbids it:

* Create job
* Create data feed
* Open job
* Start data feed

This is not a final list of the restrictions we want to have in place if the license forbids ML access but it serves as a starting point and can be easily updated following further discussions.

The change also moves the `transportClientMode` check to `createComponents` so we don’t try to start the server parts of machine learning (job manager, connection to named pipes etc.) if we are running in a transport client.

Original commit: elastic/x-pack-elasticsearch@6c19ebd3bc
2017-02-16 09:40:53 +00:00
Martijn van Groningen 54d57f6398 [ml] When trying to find a node to open a job take `max_running_jobs` into account instead of just trying and then failing.
Also before even creating a persistent task for the job, check if there ml nodes available to run the job, otherwise fail quickly.

relates elastic/x-pack-elasticsearch#545

Original commit: elastic/x-pack-elasticsearch@63fd72e56a
2017-02-16 09:58:44 +01:00
Zach e441e0dc4e [ML] Fix order that DBQ is constructed
Also use Job.ID constant instead of string literal.

Kudos to @dimitris-athanasiou for this one :)

Original commit: elastic/x-pack-elasticsearch@906b080110
2017-02-15 15:53:45 -05:00
Martijn van Groningen 6d79210f79 [ml] Registered named writable entry for DatafeedState.
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.

Original commit: elastic/x-pack-elasticsearch@89e4875f6c
2017-02-15 21:38:19 +01:00
jaymode c8b5be186d remove signing of authentication information
Now that TLS is required for node to node communication, we no longer need to sign the
authentication information to prevent tampering.

Original commit: elastic/x-pack-elasticsearch@1f86cf2395
2017-02-15 12:55:24 -05:00
Zachary Tong 65cd119733 [ML] Fix injection of DomainSplit, and only inject for Painless
Was accidentally injecting the script object, not the string version of the code.  Also
added a check so we only inject for Painless scripts (and not groovy, etc).

Minor style tweaks too.

Original commit: elastic/x-pack-elasticsearch@58c7275bd8
2017-02-15 09:31:16 -05:00
Martijn van Groningen 76ea6444cf [test] if enabled always specify setting other randomly specify setting to false.
Original commit: elastic/x-pack-elasticsearch@eacdb979e2
2017-02-14 22:28:43 +01:00
Zachary Tong cb29cbd8a9 [ML] Allow result indices to be shared. (elastic/x-pack-elasticsearch#555)
Essentially an update to https://github.com/elastic/prelert-legacy/pull/736 .  Still does not
default to using shared indices, but adds the capability for two jobs to share the same one
without conflict

Still does not default to using shared indices, just adds the capability for two jobs to share the same one without conflict.

Original commit: elastic/x-pack-elasticsearch@60d93a06ea
2017-02-14 14:59:28 -05:00
Zachary Tong 72ef1acdc0 Move domainSplit injection to ScrollDataExtractor (elastic/x-pack-elasticsearch#554)
By injecting immediately before use, we don't have to store it in the config and return
huge blobs to the user when they request the config

Original commit: elastic/x-pack-elasticsearch@2e53c930ea
2017-02-14 12:26:21 -05:00
Adrien Grand ac8f3ffec4 Fix security source filtering. (elastic/x-pack-elasticsearch#561)
Source filtering is currently written as if the fact that an object is accepted
means that all sub fields and objects are implicitly accepted, which is not
true.

Original commit: elastic/x-pack-elasticsearch@afc4440e7a
2017-02-14 17:44:20 +01:00
Martijn van Groningen 6e411c6118 [ml] Added allocation enabled node attribute setting
This node attribute setting defaults to true when ml is enabled and jobs are only opened on nodes where the allocation enabled node attribute is set to true.

Original commit: elastic/x-pack-elasticsearch@fd3f1b3058
2017-02-14 16:58:08 +01:00
Colin Goodheart-Smithe 6183015639 Adds licensing to machine learning feature (elastic/x-pack-elasticsearch#559)
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.

Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`

Original commit: elastic/x-pack-elasticsearch@48ea9d781b
2017-02-14 15:47:37 +00:00
Martijn van Groningen 67fe584e0f [ml] Also allow closing failed jobs
Relates to elastic/x-pack-elasticsearch#545

Original commit: elastic/x-pack-elasticsearch@1c814afbf0
2017-02-14 15:38:46 +01:00
Dimitris Athanasiou 4976feb53b [ML] Extract MlInitializationService steps into respective methods (elastic/x-pack-elasticsearch#560)
Original commit: elastic/x-pack-elasticsearch@c43d02e752
2017-02-14 14:05:39 +00:00
David Kyle 27a90f4f0d Remove the job ignore_downtime option (elastic/x-pack-elasticsearch#550)
Original commit: elastic/x-pack-elasticsearch@5a7c4fdfea
2017-02-14 09:43:25 +00:00
Jason Tedor c4be485514 Add support for dispatching bad requests
This commit is a response to an upstream change in Elasticsearch which
added a bad dispatch request handler.

Relates elastic/x-pack-elasticsearch#553

Original commit: elastic/x-pack-elasticsearch@c28baee799
2017-02-13 17:39:44 -05:00
jaymode 82a80dbf42 Update seriailization after backport of content type changes
Original commit: elastic/x-pack-elasticsearch@0ef4dda41d
2017-02-13 13:18:56 -05:00
Martijn van Groningen 5eb8414215 [ml] Specify the minimum supported version,
so that we don't send ml metadata to older nodes.

Original commit: elastic/x-pack-elasticsearch@f196b5c461
2017-02-13 18:15:48 +01:00
Tanguy Leroux a6e67fe51e Monitoring: remove collectors setting elastic/elasticsearch#4957
This commit removes the undocumented, test used only, setting xpack.monitoring.collection.collectors. It should not be used by users at all, and has been initially created for collector tests.

From  elastic/elasticsearch#4957

Original commit: elastic/x-pack-elasticsearch@0fa39a94b5
2017-02-13 16:36:59 +01:00
Martijn van Groningen fbcac2710a [ml] changed start datafeed validation
Original commit: elastic/x-pack-elasticsearch@266d360cae
2017-02-13 14:04:32 +01:00
Dimitris Athanasiou 3e43b591df [ML] Correctly return job stats for _all (elastic/x-pack-elasticsearch#540)
Original commit: elastic/x-pack-elasticsearch@a286ea36e9
2017-02-13 09:10:35 +00:00
Jason Tedor 1b4fec642c Enforce Java version
The Elastic Secrets vault is served via HTTPS with a Let's Encrypt
certificate. The root certificate that cross-signed the Let's Encrypt
certificates were not trusted by the JDK until 8u101. This commit adds a
version check at the start of the build to make it clear the cause of
the underlying issue, and what the fix is.

Relates elastic/x-pack-elasticsearch#541

Original commit: elastic/x-pack-elasticsearch@6bf8076cb6
2017-02-11 12:12:26 -05:00
Ryan Ernst 2571921605 Rename x-pack project names to new names with split repo
Original commit: elastic/x-pack-elasticsearch@5a908f5dcc
2017-02-10 11:02:42 -08:00
Ryan Ernst 1fb742a0ad Rename core plugin and transport dirs
Original commit: elastic/x-pack-elasticsearch@1844685f77
2017-02-10 11:02:42 -08:00