Prior to this change the integration tests could fail for up to 24
hours after a change to the C++ output format
Original commit: elastic/x-pack-elasticsearch@b8d9fa0adb
Rename model_debug_output to model_plot
The model debug output is 1 result type of the autodetect (anomaly detection) process reporting bounds/statistics/state - not anomalies - It's main usage as of time of writing is visualization in the UI, naming the 'blue shadow' under the result graph. Rename in order to remove 'debug' in the name
Original commit: elastic/x-pack-elasticsearch@86de428512
This commit restricts custom role providers to only apply to those
instances of x-pack with a platinum level license. All other license
types will not be allowed to use custom role providers. Any custom role
providers implemented via the XPackExtension will not take effect unless
the license is platinum.
relates elastic/x-pack-elasticsearch#720
Original commit: elastic/x-pack-elasticsearch@4fc35494ee
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.
Original commit: elastic/x-pack-elasticsearch@0b23ef528f
When adding support for rest request filtering of sensitive content, the overridden rest request
did not properly delegate the #getRemoteAddress method to the wrapped request. This resulted in a
NPE when a filtered rest request was created and an audit record needed to be generated.
relates elastic/x-pack-elasticsearch#714
Original commit: elastic/x-pack-elasticsearch@710b43355b
User signing was removed with the move to TLS only transport and this change removes a no longer
used setting.
Original commit: elastic/x-pack-elasticsearch@c221f1f4a4
* Changed ML action names to allow distinguishing of admin and read-only actions
using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
InternalClient when searching for results so that index/document level permissions
applied to ML results are respected
Original commit: elastic/x-pack-elasticsearch@eee800aaa8
This commit removes an unnecessary String creation from the char[] of a password and instead uses
a byte[] that is cleared after it is used to prevent the password bytes from sticking around in
memory longer than required.
Original commit: elastic/x-pack-elasticsearch@1154a68965
- introduce builtin user 'beats_system'
- init monitoring-beats ES mapping
- add beats to MonitoredSystem + Resolver
Original commit: elastic/x-pack-elasticsearch@4d7b45d54d
DataStreamDiagnostics
DataStreamDiagnostics analyzes input data regarding machine learning fit. It checks whether data is sane/plausible as anomaly detection on broken data (or misconfiguration).
Original commit: elastic/x-pack-elasticsearch@2f37d3c960
This commit fixes the SHA hashes for the Netty dependencies after Netty
was upgraded from version 4.1.8 to version 4.1.9.
Original commit: elastic/x-pack-elasticsearch@ca4c376886
This integration tests only checks if date math index names are
supported. This is nothing that a watcher test should check for.
Original commit: elastic/x-pack-elasticsearch@8a06988fff
Now machine learning is simply a feature of X-Pack, so I have made the
MachineLearning class more like the other feature classes. It no longer extends
Plugin and its createComponents() method takes an InternalClient rather than
a generic Client.
Original commit: elastic/x-pack-elasticsearch@704860147c
This commit adds an end-point to force deletion of expired data:
DELETE /_xpack/ml/_delete_expired_data
A few other things are changed too:
- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
visibility
- Commit results when quantiles are persisted to ensure they are visible
for renormalization
Original commit: elastic/x-pack-elasticsearch@8ca5272a94
Switched the order of the local atomic flag check and cluster state check,
based on the observation that we used to sometimes get a storm of cluster
state updates on initial startup, due to a race between checking for the
presence of an index template and starting to create one.
Original commit: elastic/x-pack-elasticsearch@7ae83648ce
* [ML] Wait for templates to be installed before running ML Integ tests
* Revert put job template check and preserve template change
* Review comments
Original commit: elastic/x-pack-elasticsearch@21deb34f4a
Security API to determine which (if any) of a specified set of index/cluster privileges are held by the current (runAs) user.
Intended for use by Kibana to distinguish between read/write and read-only users, but should be applicable to other uses cases also.
Closes: elastic/x-pack-elasticsearch#282
Original commit: elastic/x-pack-elasticsearch@8b4cfdb858
The only tricky thing here was what to do about ES jars. We now ignore them for the jar sha checking, and the base elasticsearch license and notice is copied here, and elasticsearch jars are mapped to those license/notice files.
Original commit: elastic/x-pack-elasticsearch@a6373cfe4e
This fixes returning results only for the job asked
when a shared index is used.
The commit also refactors the field count check to
use the field mappings API and solves a bug where the
check blows due to the _default_ type not having properties.
[Zach] Minor test tweak to MlJobIT
Original commit: elastic/x-pack-elasticsearch@729f886879
The yaml test runner now throws error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which was previously ignored. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.
Original commit: elastic/x-pack-elasticsearch@920201207c
Only distributed ml tests, only enable ml as security, watcher, monitoring etc. only add noise to the tests.
Disable mock big arrays for distributed ml tests, as nodes starting/stopping in tests somehow leaves allocations around,
which only is a test problem.
Original commit: elastic/x-pack-elasticsearch@5ff1e69036
This moves the index structure to using a single, shared index
(.ml-anomalies-shared). Custom indices can still be used by manually
setting `results_index`.
An alias is always created which points from `.ml-anomalies-<jobid>`
to `.ml-anomalies-shared`.
User defined indices are prepended with "custom-"
Index helper functions have been renamed to make this clear. Furthermore,
accessing an index should always be done either by fetching the
currently configured index/alias from the state, or using the preconfigured
alias. Because the user can specify a custom physical index, it is
impossible to determine the physical index "by convention" now.
The helpers have been configured to reflect that.
Original commit: elastic/x-pack-elasticsearch@a5368eb230
This commit adds the ability for x-pack extensions to optionally
provide custom roles providers, which are used to resolve any roles
into role descriptors that are not found in the reserved or native
realms. This feature enables the ability to define and provide roles
from other sources, without having to pre-define such roles in the security
config files.
relates elastic/x-pack-elasticsearch#77
Original commit: elastic/x-pack-elasticsearch@bbbe7a49bf
As part of authentication, we use a iterating action listener to perform asynchronous authentication against the realm
chain. When this listener is called with a response or a failure, it could be called from a thread that is not owned by
the Elasticsearch threadpool such as a LDAPConnectionReader thread. When this happens, we need to ensure that the
ThreadContext is not left with items in it otherwise we leave behind things like Authentication and hit obscure errors.
This commit stores the context when the listener calls the consumer or onResponse/onFailure is invoked, which prevents
us from polluting a external thread's ThreadContext.
Original commit: elastic/x-pack-elasticsearch@0f50fb6c10
Some json builders in the codebase were not closed. even
though this is not needed for the BytesStreamOutput being used,
there is more closing logic in the jackson classes, which we
should not rely on, that those never change or are ok to not
close.
Original commit: elastic/x-pack-elasticsearch@05a43d80ff
This is the Elastic convention, and also makes it clearer where the
actual log message from the C++ starts
Original commit: elastic/x-pack-elasticsearch@dc9aeefb08
in case of tribe node client do not even bother to check for native process version, because it isn't running.
Original commit: elastic/x-pack-elasticsearch@e6fdd5e82a
[ML] * Add MachineLearningTemplateRegistry class
[ML] * Add blocking method to put templates required by tests
[ML] * Add version check for templates
[ML] * Review comments
Original commit: elastic/x-pack-elasticsearch@07d315e56d
This norelease can be removed, because we no longer create indices automatically when missing, but use index templates.
Original commit: elastic/x-pack-elasticsearch@b6880ce7b7
Also included assignment explanation to both job and datafeed stats apis and
included executor node to datafeed stats api
Original commit: elastic/x-pack-elasticsearch@783bc77ef6
When the datafeed indexes are concrete indexes, the search
will throw IndexNotFoundException, thus a notification is being
generated. However, when the indexes are patterns, the search
does not fail.
This commit adds a warning when a lookback-only datafeed
retrieves no data as it is certain that the user misconfigured
something and it's useful to make him aware of it.
Original commit: elastic/x-pack-elasticsearch@e3490ea978
If any format other than json is sent, it should get converted to json, as watcher works under the assumption that the script is a template, hence json.
Original commit: elastic/x-pack-elasticsearch@554bcd306b
Fixeselastic/prelert-legacy#803
On Windows the JVM hangs during shutdown if the native controller process
is still running. A thread dump suggests this is because of a finalizer being
blocked. We can avoid the problem by explicitly telling the controller to exit
when the node is stopped. Although the hang is Windows-specific, it doesn't
hurt to do this on *nix too.
Original commit: elastic/x-pack-elasticsearch@6fa1c2b966
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners
This fix was mistakenly removed in another commit
Original commit: elastic/x-pack-elasticsearch@6305456d5a
Adds a preview end-point: `/_xpack/ml/datafeeds/{datafeed_id}/_preview`
The endpoint returns an array with the JSON records that will reach
writer to the c++ process. Thus, the preview can be used to verify
that a datafeed is configured correctly with regard to itself and
its corresponding job.
Original commit: elastic/x-pack-elasticsearch@1e0e9c906d
* [ML] Index template for notifications, meta and state indices
* Add ignore_unavailable option to searches so missing indices don’t throw errors
* Test for job existence before returning results
* Template mapping for results indices
* Fix tests
* Reinstate merge conflict dropped code
* Fix and tidy up JobProviderTests
Tests had badly named & duplicated tests
* Don’t check if the job indices already exist when allocating
Indices are created on demand from templates
* No need to create indices in initialisation service
Using templates now
* Remove unused code and add comment
Original commit: elastic/x-pack-elasticsearch@1ba115d4fc
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners
Original commit: elastic/x-pack-elasticsearch@a7aff45571
Refactors xcontent serialization of Request and Status to use their writable names instead of action name. That simplifies the parsing logic, allows reuse of the same status object for multiple actions and is consistent with how named objects in xcontent are used.
Original commit: elastic/x-pack-elasticsearch@3d059ea952
The DnRoleMapper was changed when moving to asynchronous authentication, which introduced a regression that leads to a
NPE being thrown on file refresh. The cause is the addition of a null listener from the LdapRealm upon construction.
This change removes the listener constructor argument as it was always null or a empty runnable and adds a check in the
addListener method to ensure the listener is not null. Finally, the DnRoleMapperTests had a bug where the temporary
filenames did not have the correct suffix and tests were not actually testing what was intended.
relates elastic/x-pack-elasticsearch#608
Original commit: elastic/x-pack-elasticsearch@f47f258590
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.
This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.
This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.
Original commit: elastic/x-pack-elasticsearch@b31d144597
In rare cases, the CertificateToolTests#testPromptingForInstanceInformation could try to use the same name multiple
times. This causes the test to fail with a message for an unknown prompt. This commit ensures no duplicates are added.
relates elastic/x-pack-elasticsearch#597
Original commit: elastic/x-pack-elasticsearch@ab8bb7ad50
This commit allows persistent actions to indicate why a task was or wasn't assigned to a certain node.
Original commit: elastic/x-pack-elasticsearch@cdacf9b10f
Changed `ProblemTracker#updateEmptyDataCount(boolean)` into `ProblemTracker#reportEmptyDataCount()`.
and added `ProblemTracker#reportNoneEmptyCount()` to reset the empty count counter.
Original commit: elastic/x-pack-elasticsearch@c1a44d6fd3
This allows it to be disabled/enabled via job updates.
It also simplifies it by removing bounds_percentile as it currently
rarely makes sense to set the bounds to another value than 95.0.
Original commit: elastic/x-pack-elasticsearch@c27fce2d86
This PR introduces a service component that is scheduled to run
once per day, around 30 minutes past midnight, local time.
Listeners can be subscribed to the service in order to get notified
when the next service trigger occurs so that they can execute
management tasks.
Currently, there are 2 such listeners: one is performing removal
of results and the other of model snapshots that have expired
their retention period.
Finally, note that the service is only installed on the master
node to ensure there is only one running on the cluster.
Original commit: elastic/x-pack-elasticsearch@c8edb6016a
Let LicensingTests.testNativeRealmMigratorWorksUnderBasicLicense wait for 3s (instead of 1s) as it is occasionally failing.
Also added logging so that if/when it does fail we can see how long we were waiting and what was taking place.
Original commit: elastic/x-pack-elasticsearch@22882ec412
Added missing named xcontent entries, tweaked start datafeed validation logic and fixed status serialization issue that were uncovered by this test.
Original commit: elastic/x-pack-elasticsearch@8b3fb012df
* Wrap exceptions if they are not ElasticsearchExceptions in TransportJobTaskAction
* Review comments
Original commit: elastic/x-pack-elasticsearch@07e900ffaa
Remove `node.attr.xpack.ml.allocation_enabled` node attribute and instead use `node.attr.max_running_jobs` attribute to indicate a node is a ml node.
(in addition to indicating how many jobs can hosted on a node)
Original commit: elastic/x-pack-elasticsearch@7979bc55b4
Fields were missing from the job's copy constructor.
This was the cause of intermittent test failures.
Original commit: elastic/x-pack-elasticsearch@3cd705ab3c
...correctly update the job's model_snapshot_id whenever
a new model snapshot is persisted. Load the snapshot that
corresponds to the job's model_snapshot_id when a job is
opened.
This PR makes a series of changes that completes the change
of having job's active model snapshot persisted on the job.
Original commit: elastic/x-pack-elasticsearch@c2e23fa1ee
A job allocation is either a job task in OPENING state or a job task that has been assigned to an executor node, but the executor node hasn't had the opportunity to set the job task status to OPENING.
In order to keep track of restarted tasks, `allocationIdOnLastStatusUpdate` field was added to `PersistentTaskInProgress` class.
This will allow persistent task implementors to detect whether the executor node has changed or has been unset since the last status update has occured.
Original commit: elastic/x-pack-elasticsearch@b7b85a8274
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.
Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.
Original commit: elastic/x-pack-elasticsearch@29d336a008
All validation should be done in the builder and the Job object should just be a dumb, immutable blob.
Original commit: elastic/x-pack-elasticsearch@a78e6edbe9
* Update running process from cluster state update
* Put a big lock on the UpdateProcessAction
* Protect concurrent update of the process with a semaphore
* Review comments
Original commit: elastic/x-pack-elasticsearch@d08b53247a
Martijn van Groningen
Dimitrios Athanasiou
David Roberts
Hendrik Muhs
David Kyle
Ali Beyad
Tanguy Leroux
Ryan Ernst
Jay Modi
Steffen Siering
Jason Tedor
Alexander Reelsen
Tim Vernum
Luca Cavanna
Zachary Tong
Igor Motov
javanna
Colin Goodheart-Smithe
jaymode
Tim Vernum
polyfractal