This is in preparation of introducing a write alias.
It adjusts all requests to persist results to do so
using a method that returns the write alias (even though
it currently returns the same as the read alias).
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@1358dd8dcf
This commit updates the RunAsIntegTests to randomly assign the superuser role to the user that
is authenticating with the cluster but the request is being run as a different user. This provides
additional validation that the authorization errors are actually coming from the user the request
is running as and not due to the authenticating user's privileges.
Original commit: elastic/x-pack-elasticsearch@c6360d13e6
Each job introduces new fields to the results index matching
the analysis terms. When the job is created, mappings for those
are added explicitly. However, when rollover is introduced, that
will not be the case. This commit prepares for that by adding
dynamic mapping of new fields as keyword.
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@8f6cd09a71
This was the behaviour in Shield 2.x, but it was accidentally changed during migration to X-Pack 5.x
Original commit: elastic/x-pack-elasticsearch@de0bf5e688
It just wastes 20 seconds while we timeout trying to open named pipes that cannot
possibly have been created.
Original commit: elastic/x-pack-elasticsearch@4e447874f6
fixes build errors. Still ensures that the timestamp is set to 'now' if the parsed logfile misses it.
Original commit: elastic/x-pack-elasticsearch@cf60e8d76b
If invalid job configs are transported to the master node then the root
cause of the validation exception gets reported as a remote_transport_exception,
which is extremely confusing.
This commit moves the validation of job configurations to the first node that
handles the action.
Fixeselastic/x-pack-kibana#1172
Original commit: elastic/x-pack-elasticsearch@5ed59d2a6f
Before searching for documents in monitoring indices, we need to ensure
that they exist and are available.
Original commit: elastic/x-pack-elasticsearch@29db55a1fe
This commit adds a base rest handler for security that handles the license checking in the security
apis. This was done previously in some rest handlers but not all and actually had issues where a
value would be returned but we may not have consumed all of the request parameters, which could
lead to a different response being returned than what we would have expected.
relates elastic/x-pack-elasticsearch#1236
Original commit: elastic/x-pack-elasticsearch@2f1100b64a
Adds defenseagainst broken scrolls to the fetching roles and users.
While Elasticsearch *shouldn't* send broken scrolls it has done so
in the past and when it does this causes security to consume the
entire heap and crash. This changes it so we instead fail the request
with a message about the scroll being broken.
Relates to elastic/x-pack-elasticsearch#1299
Original commit: elastic/x-pack-elasticsearch@dfef87e757
Some JDKs do not support the ECDSA cipher suites that we use in the EllipticCurveSSLTests, which
is the underlying cause of some CI failures. This change ensures there is at least one enabled
ECDSA cipher before testing that a connection can be made.
relates elastic/x-pack-elasticsearch#1278
Original commit: elastic/x-pack-elasticsearch@f6c93d776c
* [ML] Not an error to close a job twice
* Error if job is opening
* Address review comments
* Test closed job isn’t resolved
Original commit: elastic/x-pack-elasticsearch@7da7b24c08
Originally we used to try to get the native code version even when ML was disabled.
However, this has proved to be an annoyance in cases where people running on
unsupported platforms have disabled ML. This commit means we'll only try to get
the native code version if ML is enabled on a node.
Original commit: elastic/x-pack-elasticsearch@778d6708d2
This removes the "node" type from `.monitoring-data-2`. This data is sent to _both_ the time-based and non-time-based indexes for Elasticsearch, but the UI only used the time-based variant already.
This is another step in the process of removing the `.monitoring-data-2` index. There is now only one `_type` left in that index: `cluster_info`, which is used by the UI and phone home stats because it contains the license details _and_ the `stack_stats` (e.g., `xpack_usage`).
Original commit: elastic/x-pack-elasticsearch@2cadb5939d
This is a task towards allowing rollover.
Multiple model_size_stats are stored in order to allow
analytics of the memory usage over time. The job _stats
need to display the latest model_size_stats. Before this
commit, the latest model_size_stats was being stored with
a special ID and it was retrieved using that ID. This
does not lend itself well for rollover as we would end up
with multiple of those special IDs in the rolled indices.
This commit removes the need to store a special model_size_stats
version. Job _stats now retrieve the model_size_stats by searching
for the latest one. It also uses a manual ID for all model_size_stats
in order to maintain a single document per log_time.
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@b2796e9b08
Now that the Monitoring UI no longer checks the `.monitoring-data-2` index
for Kibana, Logstash, or Beats data, we can stop accepting the duplicated
data in that index (the _exact_ same documents are also indexed into the
time-based index for each product).
This ignores rather than rejects requests that contain such documents to
allow older clients to communicate with a 5.5+ monitoring cluster.
Original commit: elastic/x-pack-elasticsearch@def472cf2e
- role_mapping.native is always present, but contains no entries if the security index is unavailable
- file based role mapping does not allow duplicate keys
Original commit: elastic/x-pack-elasticsearch@734cf7e2c0
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.
When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.
Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.
Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
In some cases (based on the randomisation) the primary-realm would have no mappings, but the secondary-realm would.
If this occurred we wouldn't write out any mappings and the secondary-realm would not authorize the access being tested
Original commit: elastic/x-pack-elasticsearch@8e81ec9dd7
This introduces a role-mapping API to X-Pack security.
Features:
- A `GET`/`PUT`/`DELETE` API at `/_xpack/security/role_mapping/`
- Role-mappings are stored in the `.security` index
- A custom expression language (in JSON) for expressing the mapping rules
- Supported in LDAP/AD and PKI realms
- LDAP realm also supports loading arbitrary meta-data (which can be used in the mapping rules)
- A CompositeRoleMapper unifies roles from the existing file based mapper, and the new API based mapper.
- Usage stats for native role mappings
Original commit: elastic/x-pack-elasticsearch@d9972ed1da
This is related to elastic/elasticsearch#24412. That commit changed how
ListenableActionFuture implementations are created. This commit
updates x-pack to be compatible with those changes. In particular, all
the usages of ListenableActionFuture in x-pack could be replaced with
PlainActionFuture as the "listening" functionality was not being used.
Original commit: elastic/x-pack-elasticsearch@7c8d8e3df9
JobDataDeleter handles the deletion logic for 3 cases:
1. deleting a model snapshot and its state docs
2. deleting all results after a timestamp
3. deleting all interim results
The last 2 are currently implemented by manually performing
a search and scroll and then adding matching hits in a bulk
delete action. This operation is exactly what delete-by-query
does.
This commit changes JobDataDeleter to use delete-by-query. This
makes the code simpler and less error-prone. The downside is
losing some logging which seems non-critical. Unit tests for
JobDataDeleter are also removed as they are heavily mocked tests,
adding little value and high maintenance cost. This functionality
is tested by integration tests already.
relates elastic/x-pack-elasticsearch#821
Original commit: elastic/x-pack-elasticsearch@7da91332bd
The distribution of watches now happens on the node which holds the
watches index, instead of on the master node. This requires several
changes to the current implementation.
1. Running on shards and replicas
In order to run watches on the nodes with the watches index on its
primaries and replicas. To ensure that watches do not run twice, there is
a logic which checks the local shards, runs a murmurhash on the id and
runs modulo against the number of shards and replicas, this is the way to
find out, if a watch should run local. Reloading happens
2. Several master node actions moved to a HandledTransportAction, as they
are basically just aliases for indexing actions, among them the
put/delete/get watch actions, the acknowledgement action, the de/activate
actions
3. Stats action moved to a broadcast node action, because we potentially
have to query every node to get watcher statistics
4. Starting/Stopping watcher now is a master node action, which updates
the cluster state and then listeners acts on those. Because of this watches
can be running on two systems, if you those have different cluster state
versions, until the new watcher state is propagated
5. Watcher is started on all nodes now. With the exception of the ticker
schedule engine most classes do not need a lot of resources while running.
However they have to run, because of the execute watch API, which can hit
any node - it does not make sense to find the right shard for this watch
and only then execute (as this also has to work with a watch, that has not
been stored before)
6. By using a indexing operation listener, each storing of a watch now
parses the watch first and only stores on successful parsing
7. Execute watch API now uses the watcher threadpool for execution
8. Getting the number of watches for the stats now simply queries the
different execution engines, how many watches are scheduled, so this is
not doing a search anymore
There will be follow up commits on this one, mainly to ensure BWC compatibility.
Original commit: elastic/x-pack-elasticsearch@0adb46e658
Within the same JVM, setting the number of processors available to Netty
can only be done once. However, tests randomize the number of processors
and so without intervention would attempt to set this value multiple
times. Therefore, we need to use a flag that prevents setting this value
in tests.
Relates elastic/x-pack-elasticsearch#1266
Original commit: elastic/x-pack-elasticsearch@d127149725
Many of the tests assume that the trial license has already been generated before the test gets to run. As this is asynchronously triggered upon node
startup, however, there is no guarantee that trial license generation has completed before the tests get to execute, leading to null values when
checking clusterService.state().metaData().custom(LicensesMetaData.TYPE).
Original commit: elastic/x-pack-elasticsearch@d909c9ba95
LicenseManagerServiceTests sometimes fails in jenkins, but fairly
rarely. We don't get useful logs when it does. This cranks up
the log level and adds some more assertions so we can better track
down where the failure comes from.
Relates to elastic/x-pack-elasticsearch#222
Original commit: elastic/x-pack-elasticsearch@3e08725fc7
When a user creates a datafeed, as well as checking they have permission
to create a datafeed we also check that they have permission to search the
indices they've configured the datafeed to search.
Previously this second check was erroneously done for the user who issued
the put_datafeed request, whereas it should be done as the runas user for
that request.
Original commit: elastic/x-pack-elasticsearch@4c35204c66
When we revert to snapshot, if we delete intervening results
we should delete with an open end on the time range for the
case when future data has been posted to the job.
Original commit: elastic/x-pack-elasticsearch@c3f5e8f19e
The one argument ctor for `Script` creates a script with the
default language but most usages of are for testing and either
don't care about the language or are for use with
`MockScriptEngine`. This replaces most usages of the one argument
ctor on `Script` with calls to `ESTestCase#mockScript` to make
it clear that the tests don't need the default scripting language.
Original commit: elastic/x-pack-elasticsearch@c1d05b7357
When the active directory realm was refactored to add support for authenticating against multiple
domains, only the default authenticator respected the user_search.filter setting. This commit moves
this down to the base authenticator and also changes the UPN filter to not include sAMAccountName
in the filter.
Original commit: elastic/x-pack-elasticsearch@d2c19c9bee
- Removes need to handle exception from action methods
- Clearly renames DatafeedJobIT to DatafeedJobsRestIT to distinguish
from DatafeedJobsIT
- Refactors DatafeedJobsIT to reuse MlNativeAutodetectIntegTestCase
Original commit: elastic/x-pack-elasticsearch@5bd0c01391
Cross cluster search uses ClusterSearchShardsAction under the covers.
Without this change, you would need both "read_cross_cluster" and "view_index_metadata" privilegs in order to have permission to execute searches from a remote cluster.
Original commit: elastic/x-pack-elasticsearch@65a6aff329
If a single permission set does not have a query defined then this should be considered as the user
not having document level security for the indices matching that pattern. However, the lack of
document level security was not being taken into account and document level security was being
applied when it should not have been.
Original commit: elastic/x-pack-elasticsearch@f5777c2f37
Many tests in monitoring use the pattern of calling first awaitMonitoringDocsCount, and then doing a search that checks certain properties, assuming
that the doc count is correct at that point. In the presence of replicas, awaitMonitoringDocsCount might not wait for all shard copies to have the
desired property. A subsequent search might then hit a shard where the property does not hold. As these tests randomize the number of replicas
(through the random_index_template), it easier to constrain awaitMonitoringDocsCount just to the primary and then do subsequent checks just by
querying the primary.
Original commit: elastic/x-pack-elasticsearch@4165beb903
This commit creates a single server socket that will be connected to by local sockets. The local
sockets will use the port of the previously stopped ldap server as the local port. This will
prevent the ldap library from establishing a connection. The previous use of server sockets for
this did not work on all operating systems as the backlog parameter has platform specific meaning.
Original commit: elastic/x-pack-elasticsearch@03b6bf39d4
This commit adds a timeout for the expiration of invalidated tokens so that we can expect that the
requests will have been finished before we do the assertions on the internal test cluster.
Original commit: elastic/x-pack-elasticsearch@2928706224
This commit adds a token based access mechanism that is a subset of the OAuth 2.0 protocol. The
token mechanism takes the same values as a OAuth 2 standard (defined in RFC 6749 and RFC 6750),
but differs in that we use XContent for the body instead of form encoded values. Additionally, this
PR provides a mechanism for expiration of a token; this can be used to implement logout
functionality that prevents the token from being used again.
The actual tokens are encrypted using AES-GCM, which also provides authentication. The key for
encryption is derived from a salt value and a passphrase that is stored on each node in the
secure settings store. By default, the tokens have an expiration time of 20 minutes and is
configurable up to a maximum of one hour.
Relates elastic/x-pack-elasticsearch#8
Original commit: elastic/x-pack-elasticsearch@3d201ac2bf
This is an issue where a bucket can have both interim results and
non-interim results, a bucket should never have both at the same time.
The steps to cause this situation are:
1. Flush a running job and create interim results
2. Close that job (this does not delete interim results)
3. Re-open the job and POST data
4. The job will eventually emit a bucket result which mingles with the
existing interim results
Normally interim results are deleted by AutoDetectResultProcessor when a
bucket is parsed following a flush command. Because of the close and
re-opening of the job AutoDetectResultProcessor no longer knows that a
previous flush command creating interim results.
The fix is to always delete interim results the first time
AutoDetectResultProcessor sees a bucket.
relates elastic/x-pack-elasticsearch#1188
Original commit: elastic/x-pack-elasticsearch@5326455f54
In the SessionFactoryLoadBalancingTests, we sometime want a connection to a certain IP and Port to
fail as a way to mock an unresponsive/disconnected LDAP server. The test does this by starting up
multiple LDAP servers and then shutting some down. When the server is shut down the port that it
was bound to is open for another process or test to bind to, which can lead to sporadic failures in
CI. This change is a best effort attempt to prevent this by binding a server socket to the port and
filling its backlog so other connections should fail.
Relates elastic/x-pack-elasticsearch#1195
Original commit: elastic/x-pack-elasticsearch@b31a560c93
The DatafeedJobsIT.testRealtime test fails from time to time.
The test seems to take a long time to execute the flush action
after the lookback. This could make sense as the test produces
a few records over the span of a week with 5 minutes bucket_span.
Thus, flush will end up doing a lot of word to create results
for so many buckets.
This change increases the bucket_span to 1 hour. Hopefully, this
will stop the failures.
Relates elastic/x-pack-elasticsearch#1162
Original commit: elastic/x-pack-elasticsearch@4366907371
This commit fixes the support for elliptic curve certificates that are specified as a PEM file.
These certificates and private keys can now be read properly and a integration test was added to
ensure that TLS also functions correctly with these certificates.
Original commit: elastic/x-pack-elasticsearch@6d6d579c88
This commit reduces spamming of the logs when a common SSL exception is encountered such as a
client not trusting the server's certificate or a plaintext request sent to a channel that expects
TLS traffic.
relates elastic/x-pack-elasticsearch#1062
Original commit: elastic/x-pack-elasticsearch@94959e79f6
This change prevents the situation where cleanup of ML indices immediately
after deleting a job leaves the audit notification in limbo because the index
it was due to be indexed into has been deleted.
Relates elastic/x-pack-elasticsearch#1142
Original commit: elastic/x-pack-elasticsearch@300e9c36ce
Ordinary Kibana users should not have access to the cluster state of ES,
and therefore they should not be able to access ML jobs without explicit
permission.
Original commit: elastic/x-pack-elasticsearch@77273d561a
When a condition is unmet, the ack status of the actions needs to be
resetted again, so that new alerts can be triggered.
Due to a bugfix this functionality was removed from ES 5.0.0-alpha5
onwards.
relates elastic/x-pack-elasticsearch#1123
Original commit: elastic/x-pack-elasticsearch@83db2cecf9
Persistent tasks should verify that completion notification is done for correct version of the task, otherwise a delayed notification from an old node can accidentally close a newly reassigned task.
Original commit: elastic/x-pack-elasticsearch@478bb6e730
* Adds a check to wait for active tasks for XPackRestIT
* uses test logger
* Change to use assertBusy instead of awaitBusy
* fixes failures with active tasks remaining
* Moves wait for pending tasks into MlRestTestStateCleaner
* remove unecessary log line
Original commit: elastic/x-pack-elasticsearch@1f098dbb64
By creating the watches via the exporter, we get to afford ourselves
with a much more automatic and simpler set of security permissions.
This does limit us in a few ways (e.g., every exporter has to deal with
cluster alerts itself, which means that newer releases of Kibana cannot
help by adding newer cluster alerts for older, still-monitored
clusters).
Original commit: elastic/x-pack-elasticsearch@448ef313c3
When Logstash 5.2 - 5.3 submit documents via the `_xpack/monitoring/_bulk`
endpoint, it sends its time-based documents with an explicit `_id` of
`""`.
This used to be automatically ignored by Monitoring, but we now accept the
_id that we are given (including `null`). ES, prior to 5.3.1, accepted
`""` as a valid `_id` through the `_bulk` endpoint, which means that it
blindly accepted and overwrote documents given that ID, meaning that all
Logstash instances "shared" the exact same document and therefore the UI
becomes useless.
This change allows `""` to be used and it simply replaces that value, and
only that value, with `null`. This enables backwards compatibility with LS
5.2 - 5.3.0.
Original commit: elastic/x-pack-elasticsearch@889578e61e
PersistentTasksCustomMetadata was using a generic param named `Params`. This conflicted with the imported interface `ToXContent.Params`. The java compiler was preferring the generic param over the interface so everything was fine but Eclipse apparently prefers the interface int his case which was screwing up the Hierarchy and causing compile errors in Eclipse. This changes fixes it by renaming the Generic param to `P`
Original commit: elastic/x-pack-elasticsearch@8528870684
- Mark all security indices (that is all indices managed by SecurityLifecycleService) as "superuser only" (only superuser role can have direct permissions)
- Add unit tests for IndexLifecycleManager
Original commit: elastic/x-pack-elasticsearch@e4478825e0
This commit removes the SecuredString class that was previously used throughout the security code
and replaces it with the SecureString class from core that was added as part of the new secure
settings infrastructure.
relates elastic/x-pack-elasticsearch#421
Original commit: elastic/x-pack-elasticsearch@e9cd117ca1
When a index name pattern contains both date math and wildcards, the name resolution does not
return the expected result. This change moves the date math resolution to before our attempts to
match wildcards so that both can be used in the same pattern.
relates elastic/x-pack-elasticsearch#1065
Original commit: elastic/x-pack-elasticsearch@9f48b42fad
let close job and stop datafeed apis redirect to elected master node.
This is for cluster state observation purposes, so that a subsequent open and then close job or
start and then stop datafeed see the same local cluster state and sanity validation doesn't fail.
Original commit: elastic/x-pack-elasticsearch@21a63184b9
introduced separate task names to register the persistent tasks executors and params.
Also renamed start and stop datafeed action names to be singular in order to be consistent with open and close action names.
Original commit: elastic/x-pack-elasticsearch@21f7b242cf
Support for default settings has been removed in core and so some
methods were refactored. This commit responds to this change in core.
Original commit: elastic/x-pack-elasticsearch@b22c612de4
The path has changed so it’s no longer possible to distinguish between data feed and job tasks.
The preceding test get_datafeed provides ample coverage anyway.
Original commit: elastic/x-pack-elasticsearch@780b1beb6b
When the execute watch API is called without recording the execution
in the watch history, the watch status is not updated, in order to not
divert the in-memory object status and the one persisted on disk.
In order to work around this issue, the execute watch API can simply
clone a new watch status and a new watch, which means the object in
the watch store is never updated. This allows for execution and changing
of the watch status, before it is returned to the client.
relates elastic/x-pack-elasticsearch#889
Original commit: elastic/x-pack-elasticsearch@6a0d9c9a78
Changes persistent task serialization and forces params and status to have the same writeable name as the task itself.
Original commit: elastic/x-pack-elasticsearch@59cf3dca39
remove `node.attr.max_running_jobs` node attribute and use `node.attr.ml.enabled` node attribute instead to know whether a node is a ml node or not.
Also renamed `max_running_jobs` setting to `xpack.ml.max_running_jobs`.
Original commit: elastic/x-pack-elasticsearch@798732886b
Removes the last pieces of ActionRequest from PersistentTaskRequest and renames it into PersistTaskParams, which is now just an interface that extends NamedWriteable and ToXContent.
Original commit: elastic/x-pack-elasticsearch@5a298b924f
Following this change, if the user runs on a platform that we don't ship
ML binaries for:
* If ML is enabled the node still refuses to start, but clearly says why
* If ML is disabled the node starts up without logging any errors
Original commit: elastic/x-pack-elasticsearch@af4fb8c411
Now that task id are strings instead of longs (elastic/x-pack-elasticsearch#1035), ml can use the job and datafeed as task id.
This removes logic that would otherwise iterate over all tasks and check if the task's request id was equal to the provided id and instead just do lookup in the task map.
Job and datafeed task ids are prefixed with either 'job-' or 'datafeed-', because job and datafeed ids don't have to be unique as they are stored separately from each other.
Original commit: elastic/x-pack-elasticsearch@b48c2b368a
This built-in watcher_admin role is able to execute all watcher actions,
read the watch history indices and read the watches index
index. The watcher_user role allows to GET a watch and to get the stats and thats it.
relates elastic/x-pack-elasticsearch#978
Original commit: elastic/x-pack-elasticsearch@11b33a413b
- stops the datafeed when post/flush throw a conflict exception.
A conflict exception signifies the job state is not opened, thus
we are better off stopping the datafeed.
- handles flushing the job the same way as posting to the job.
relates elastic/x-pack-elasticsearch#855
Original commit: elastic/x-pack-elasticsearch@49a54912c2
Makes the log more readable in editors not set to UTF-8.
Customers may well be in this situation on Linux/Windows.
Original commit: elastic/x-pack-elasticsearch@4e59fc90cf
The commit changes how LocalExporterTests stops: it now uses the
node_stats document collected on each node and check if it's older
than a given number of seconds (10). It also removes log traces.
Original commit: elastic/x-pack-elasticsearch@0384690b41
Before this change the persistent task operations related to opening
and closing jobs would time out a long time before the operations
related to native processes.
Original commit: elastic/x-pack-elasticsearch@23076b773b
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"
Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request.
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.
This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
Closes: elastic/x-pack-elasticsearch#887
Original commit: elastic/x-pack-elasticsearch@b81b363729
The PR detects if SMILE is being provided, then correctly slices the stream such that each document is parsed individually. This is required because jackson's SMILE parser is stricter than it's JSON parser and will stop parsing when it hits a streamSeparator (unlike JSON, which will eagerly try to find more objects to parse).
Removes the forced-headers from the various REST tests.
relates elastic/x-pack-elasticsearch#642
Original commit: elastic/x-pack-elasticsearch@c0e97cd545
Instead of having a separate listener for indicating that the current task is finished, this commit is switching to use allocated object itself.
Original commit: elastic/x-pack-elasticsearch@7ad5362121
`PersistentTasksExecutor#getAssignment(...)` should be a cheap and side-effect free method,
but in case of `OpenJobPersistentTasksExecutor` and `StartDatafeedPersistentTasksExecutor` before this change it would index a document each time `getAssignment(...)` was invoked
Original commit: elastic/x-pack-elasticsearch@5ca5890baf
The change applies chunking by default on aggregated datafeeds.
The chunking is set to a manual mode with time_span being
1000 histogram buckets.
The motivation for the change is two-fold:
1. It helps to avoid memory pressure/blowing.
Users may perform a lookback on a very long period of time. In that
case, we may hold a search response for all that time which could
include too many buckets. By chunking, we avoid that situation
as we know we'll only keep results for 1000 buckets at a time.
2. It makes cancellation more responsive.
In elastic/x-pack-elasticsearch#862 we made the processing of a search response cancellable in a
responsive manner. However, the search phase cannot be cancelled at
the moment. Chunking makes the search phase shorter, which will
result to a better user experience when they stop an aggregated
datafeed.
Also note the change sets the default chunking_config on datafeed
creation so the setting is no longer hidden.
Relates to elastic/x-pack-elasticsearch#803
Original commit: elastic/x-pack-elasticsearch@ae8f120f5f
When a datafeed task is created but it cannot be assigned the task
has a null status. This means _stats report it as stopped, however
deleting it fails. In addition, it's a better experience to error
the start datafeed request all together and give the user the chance
to fix his data indices.
This change fails a datafeed-start if it cannot be assigned.
relates elastic/x-pack-elasticsearch#1018
Original commit: elastic/x-pack-elasticsearch@532288fda0
Retries should be already handled by TransportMasterNodeAction, there is no need to introduce another retry layer in Persistent Tasks code.
Original commit: elastic/x-pack-elasticsearch@967ac7f7fa
This commit changes how LocalExporterTests stops the monitoring
components: it first stops the monitoring service (but keeps the
local exporter enabled), deletes and checks if monitoring indices
are recreated, and then disables the local exporter.
Original commit: elastic/x-pack-elasticsearch@4c4809a660
Closing a job may take a while. In the meantime it is possible to start a datafeed, because before this change the job state remained OPENED.
With this change when the executor node receives the close job request, it will first set the status to CLOSING and after that closes the job (closing autodetect process, etc.).
relates elastic/x-pack-elasticsearch#990
Original commit: elastic/x-pack-elasticsearch@d8d89c0756
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.
Original commit: elastic/x-pack-elasticsearch@f1665815c2
The execution has diverged too much from post data, flush and update process apis, since the close all jobs have been added.
The logic is now easier to understand as it exist in a single source file instead of in both CloseJobAction and TransportJobTaskAction.
Original commit: elastic/x-pack-elasticsearch@daf5fabad5
Users currently have difficulty diagnosing authentication failures.
Some logging messages mislead them, and in other cases there are unexpected behaviours that are not logged at all.
These additional DEBUG log messages and change some existing messages in an attempt to alleviate that problem.
Original commit: elastic/x-pack-elasticsearch@c6ea98b038
Increase the timeout to give enough time for a datafeed to
stop smoothly.
This is the second step to avoid hitting the default timeout.
The first was ensuring aggregated datafeed is cancellable in
a responsive manner. The third and final step will be to
apply chunking in aggregated datafeeds in order to shorten
the duration of the search, which will make cancellation even
more responsive.
Relates elastic/x-pack-elasticsearch#803
Original commit: elastic/x-pack-elasticsearch@db642330ec
This commit restores the ability to build x-pack-elasticsearch without issues when running without
access to the internet. When the `--offline` flag is used, we will not try to contact vault and the
aws apis to retrieve the ml-cpp binaries but instead gradle will use a cached version even though
it may be expired.
relates elastic/x-pack-elasticsearch#726
Original commit: elastic/x-pack-elasticsearch@b0915d8fa9
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.
relates elastic/x-pack-elasticsearch#870
Original commit: elastic/x-pack-elasticsearch@34bdce6e99
This commit renames and moves the forked delete by query classes from being ml specific to being a
xpack common class since an upcoming security feature plans to make use of this. Additionally, this
commit fixes a issue where the dbq action was being executed by the calling user instead of the
xpack user for certain requests. This was found when adding a authorization change that restricts
this action's execution to the xpack user only.
Original commit: elastic/x-pack-elasticsearch@d5967e7255
There was a problem with the way CompositeBytesReference was used in the
StateProcessor. In the case of a large state document we ended up with a
deeply nested CompositeBytesReference that then caused a deep stack and N^2
processing in the bulk action processor.
This change uses an intermediate list of byte arrays that get combined into
a single CompositeBytesReference to avoid the deep nesting.
Additionally, errors in state processing now bubble up to close the state
stream, which will cause the C++ process to stop trying to persist more state.
Finally, the results processor also times out after a similar period (30 minutes)
to that used by the state processor.
Original commit: elastic/x-pack-elasticsearch@ceb31481d1
Rather than using an async call, this leverages
the Assignment logic while selecting nodes.
Now with 300% more tests!
Original commit: elastic/x-pack-elasticsearch@300d628f72
It has been observed that Amazon EBS volumes created from snapshots can
have very high latency the first time a given block is accessed. This
can lead to named pipes taking longer than 2 seconds to create.
Since the native processes create their named pipes immediately after
startup, and this only takes a fraction of a second on a local disk, 2
seconds was considered a generous timeout, but it seems that in the case
of a remote NAS with lazy provisioning it's not long enough. During
debugging a latency of just over 3 seconds was observed. The timeouts
have been increased to 10 seconds.
relates elastic/x-pack-elasticsearch#922
Original commit: elastic/x-pack-elasticsearch@c90434c948
Moves the direct management of the security index from SecurityLifecycleService to IndexLifecycleManager, so that the SecurityLifecycleService can take responsibility for several indices.
Multiple security indices are required as we move away from storing multiple types in a single index.
Original commit: elastic/x-pack-elasticsearch@fde3a42b4d
The IndexAuditTrailMutedTests have a threadpool but fail to set it on the test client, which causes
a NPE and tests to fail.
Original commit: elastic/x-pack-elasticsearch@d34a4ce080
As the snapshot that is loaded is an important operational
aspect of a job, this change adds a notification that displays
the loaded snapshot with its latest_record_timestamp and the
job's latest_record_timestamp. Having both allows us to discover
when a job is recovering after a node failure.
relates elastic/x-pack-elasticsearch#872
Original commit: elastic/x-pack-elasticsearch@c2dee495a2
The test fails on slow machines because of inflight bulk requests
that hit one node while the others are stopping. This commit adds
more time (10s), equivalent to 2 to 3 collection interval, to delete
the monitoring indices. It also add TRACE logging level for the test.
Original commit: elastic/x-pack-elasticsearch@b433937946
* [ML] Adds jobType to Job
This change adds `jobType` field to teh `Job` class so that when the job is written to the index a `job_type` field is written int he document. This will help separate this type of job from other new job types in the future so migrating the index to allow those new type of jobs will be easer
relates elastic/x-pack-elasticsearch#798
* Addresses review comments
Original commit: elastic/x-pack-elasticsearch@d9fd11edb3
When the LDAP SDK returns a SearchResult that has a non-success ResultCode, convert it to an exception and call onFailure
A configuration setting controls whether failures in referrals should be fatal (defaults to ignoring errors)
Closes: elastic/x-pack-elasticsearch#717
Original commit: elastic/x-pack-elasticsearch@4159758c2a
PersistentTasksService methods are not using ActionListener<PersistentTask<?>> instead of PersistentTaskOperationListener.
Original commit: elastic/x-pack-elasticsearch@f95d8bda3d
The `FieldPermissions` class incorrectly assumed that the `granted` and `denied` arrays were
sorted, so it could do a `binarySearch` to see if `_all` was in the arrays.
Original commit: elastic/x-pack-elasticsearch@49b5875602
This is a follow-on to elastic/x-pack-elasticsearch#939, which removes the use of Arrays.binarySearch in the FieldPermissions
class. This change removes other incorrect uses in the rest of the x-pack code and replaces them
with a stream based implementation.
Original commit: elastic/x-pack-elasticsearch@ccca7e9bad
Before this change, aggregation datafeeds used the histogram bucket
key as the record timestamp that is posted to the job. That meant
that the latest_record_timestamp at the end of a datafeed run was
the start of the latest seen histogram bucket. Upon continuing the
datafeed, the search starts from one millisecond after the
latest_record_timestamp. Hence, data may be fetched for a second time.
This change requires a max aggregation on the time_field nested in
the histogram bucket. It then reads the timestamp from that agg.
This ensures datafeed can restart without duplicating data.
relates elastic/x-pack-elasticsearch#874
Original commit: elastic/x-pack-elasticsearch@f820efa866
- include 'real-time' instead of now as the end time for real-time
datafeeds
- do not notify lookback is completed when datafeed was stopped
- do not notify datafeed switch to real-time when datafeed was stopped
Relates elastic/x-pack-elasticsearch#878
Original commit: elastic/x-pack-elasticsearch@aa22f9b86f
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.
Relates elastic/x-pack-elasticsearch#942
Original commit: elastic/x-pack-elasticsearch@a6085964d3
This commit reenables the Monitoring Bulk Api REST tests. The XPackRestIT
now enables/disables the local default exporter before executing the monitoring
tests, and also waits for the monitoring service to be started before executing
the test.
Original commit: elastic/x-pack-elasticsearch@10b696198c
State processing can take a lot longer than log processing, even after
the C++ process has closed its end of the pipe. The pipe has a buffer,
and indexing the state document(s) in that buffer can take more than a
second.
relates elastic/x-pack-elasticsearch#945
Original commit: elastic/x-pack-elasticsearch@65f5075028
* [ML] Set job create time on server
* Job.Builder serialisation tests
* Make setCreateTime package private
Original commit: elastic/x-pack-elasticsearch@d2d75e0d7b
Adds following validations:
- aggregations must contain date_histogram or histogram at the top level
- a date_histogram has to have its time_zone to UTC (or unset which
defaults to UTC)
- a date_histogram supports calendar intervals only up to 1 week
to avoid the length variability of longer intervals
- aggregation interval must be greater than zero
- aggregation interval must be less than or equal to the bucket_span
Original commit: elastic/x-pack-elasticsearch@404496a886
* Remove JobManagers dependency on JobResultsPerister
* Remove unneeded call to refresh the state index
Original commit: elastic/x-pack-elasticsearch@0b2351bba7
If jobs are being deleted then the operations required to get stats
could fail with unexpected exceptions. When stats for multiple jobs
were being requested, this would previously cause the whole operation
to fail.
This commit changes the stats endpoint to ignore jobs that are being
deleted.
Fixeselastic/prelert-legacy#837
Original commit: elastic/x-pack-elasticsearch@6ac141a987
When this happens it means the job has been deleted, which in turn means
the C++ process has been stopped, so there's no need to send it a message
and hence no problem worth logging a stack trace for.
This differs from elastic/x-pack-elasticsearch#896 because elastic/x-pack-elasticsearch#896 was for a similar situation with
closed jobs, whereas this one is for deleted jobs.
Original commit: elastic/x-pack-elasticsearch@9bb4e98fe7
The test is too rigid on checking the right number of node_stats documented that are collected. It happens if a node takes time to start, the node_stats count % numNodes will always be different than 0.
It also adds more logging for LocalBulk failures.
Original commit: elastic/x-pack-elasticsearch@1ebb20b6f6
In order to prevent tasks state updates by stale executors, this commit adds a check for correct allocation id during status update operation.
Original commit: elastic/x-pack-elasticsearch@b94eb0e863
This commit changes the LocalExporterTests so that it now test
various randomized cases in a single test. This should speed up
the test as well as minimize the failures due to multiple start
/stop of the exporter. It also uses the MonitoringBulk API
instead of calling the Exporter instances, which makes more sense
since it is the normal way to index monitoring documents.
Related elastic/x-pack-elasticsearch#416
Original commit: elastic/x-pack-elasticsearch@f8a4af15cd
Detector configs are validated both by our C++ and by our Java code.
If the C++ is stricter than the Java then error reporting is poor.
This commit adds two extra validation checks to the Java code that
were already present in the C++ validation.
relates elastic/x-pack-elasticsearch#856
Original commit: elastic/x-pack-elasticsearch@bd4ce2377c
It's possible for a C++ process to exit between the time when a
config update message for it is queued and the time that message
is processed. This commit ensures we don't spam the log with a
stack trace in this situation, as it's not a problem at all.
relates elastic/x-pack-elasticsearch#891
Original commit: elastic/x-pack-elasticsearch@81af8eaf70
Aggregated data extraction is done in 2 phases:
1. search
2. process response
The first phase cannot be currently cancelled. However, it usually
is the fastest of the two.
The second phase processes the histogram buckets in the search
response into flat JSON and then posts the result stream to the job.
This phase can be split into batches where a few buckets are posted
to the job at a time. Cancelling can then work between batches.
This commit changes the AggregationDataExtractor to process the
search response in batches. The definition of a batch is crucial
as it has to be short enough to allow for responsive cancelling,
yet long enough to minimise overhead due to multiple calls to the
post data action. The number of key-value pairs written by the
processor is a good candidate for a batch size measure. By testing,
1000 seems to be an effective number.
relates elastic/x-pack-elasticsearch#802
Original commit: elastic/x-pack-elasticsearch@ce3a172411
The native process can only handle one operation at a time, so in order the protect against multiple operation at a time (e.g. post data and flush or multiple post data operations) there should be protection in place to guarantee that at most only a single thread interacts with the native process. The current protection is broken when a job close is executed, more specifically the wait logic is broken here.
This commit changes the threading logic when interacting with the native process by using a custom `ExecutorService` that that uses a single worker thread from `ml_autodetect_process` thread pool to interact with the native process. Requests from the ml apis are initially being queued and this worker thread executes these requests one by one in the order they were specified.
Removed the general `ml` threadpool and replaced its usages with `ml_autodetect_process` or `management` threadpool.
Added a new threadpool just for (re)normalizer, so that these operations are isolated from other operations.
relates elastic/x-pack-elasticsearch#582
Original commit: elastic/x-pack-elasticsearch@ff0c8dce0b
The slack tests seem to fail periodically with not output
This commit tries to add some more verbose output by
making the query more broad and take failures into account
to uncover, what happens in this test.
Relates elastic/x-pack-elasticsearch#836
Original commit: elastic/x-pack-elasticsearch@e601b3a0df
This change adds a retain field to model snapshots.
A user can set retain to true/false via the update model snapshot API.
Model snapshots with retain set to true will not be deleted by
the daily maintenance service regardless of whether they expired.
This allows users to keep always keep certain snapshots around for
potentially reverting to in the future.
relates elastic/x-pack-elasticsearch#758
Original commit: elastic/x-pack-elasticsearch@2283989a33
* Removed OPENING and CLOSING job states. Instead when persistent task has been created and
status hasn't been set then this means we haven't yet started, when the executor changes it to STARTED we have.
The coordinating node will monitor cs for a period of time until that happens and then returns or times out.
* Refactored job close api to go to node running job task and close job there.
* Changed unexpected job and datafeed exception messages to not mention the state and instead mention that job/datafeed haven't yet started/stopped.
Original commit: elastic/x-pack-elasticsearch@37e778b585
Add the `.monitoring-alerts-2` index template via the exporter. This
avoids a very common problem where the user wipes out their monitoring
indices manually, which means that the watches would then create an index
with a dynamic mappings.
This adds a mechanism for posting a template that is not associated with a
Resolver (convenient for the forthcoming work _and_ for a future Logstash
index).
Original commit: elastic/x-pack-elasticsearch@a4cfc48191
The stopped and removeOnCompletion flags are not currently used, this commit removes them for now to simplify things.
Original commit: elastic/x-pack-elasticsearch@c636c2817e
Previously a `kill -9` on the `autodetect` process associated with a
job would leave the job in the OPENED state.
Now if the C++ process dies before a request to close the job is made
then the job state is set to FAILED.
For this purpose C++ process death is defined as end-of-file on the
log stream. (Technically it would be possible to get end-of-file on
the log stream while the C++ process was still running, but this
would also represent an unexpected and undesirable situation.)
Original commit: elastic/x-pack-elasticsearch@2b74c56a79
This to avoid to lose data counts when the job gets restarted on another node.
The job stats api returns live data counts, which may not have been persisted to an index,
so getting the data counts via search api will give us a better guarantee that when
the job gets restarted the datacounts are there too. During job restart a get call is being
done to get data counts in the order to initialize the job.
Original commit: elastic/x-pack-elasticsearch@901952da85
Previously the GET/PUT/DELETE filters actions were master node actions. This is not necessary since the filters are stored in an index rather than the cluster state. This change makes the actions extend `HandledTransportAction` so they can be run on any node.
The change also makes PutFilterAction.TransportAction use the TransportBulkAction instead of the deprecated TransportIndexAction.
relates elastic/x-pack-elasticsearch#756
Original commit: elastic/x-pack-elasticsearch@c6df04382e
This commit makes the MonitoringDoc immutable and removes the type() and id() methods from "resolvers" so that they are not anymore in charge of computing the documents types and ids. Now each MonitoringDoc knows its type and is able to compute its own id if needed.
Original commit: elastic/x-pack-elasticsearch@5161cedcc8
This commit marks the x-pack plugin as having a native controller. This
is now a requirement in core for any plugin that forks a native process
to display a warning to the user when they install the plugin.
Relates elastic/x-pack-elasticsearch#839
Original commit: elastic/x-pack-elasticsearch@3529250023
Refactors NodePersistentTask and RunningPersistentTask into a single AllocatedPersistentTask. Makes it possible to update Persistent Task Status via AllocatedPersistentTask.
Original commit: elastic/x-pack-elasticsearch@8f59d7b819
All of our code supports configuring email addresses in the
email action not only via a JSON array, but also via a comma
separated value (we also have tests for this). However in one bit
we did not support this, where an email template is rendered to
a concrete email.
This commit fixes the last piece, so that users will be able to
specify comma separated email adresses.
The main use case for this is having an array of email addresses,
that can be joined in mustache with a comma in order to send to
several recipients.
Original commit: elastic/x-pack-elasticsearch@19794ba612
This commit ensures that upon reopening a job, the in-memory
model size stats are correctly initialized from the ones
last persisted in the results index.
This fixes the bug that could be seen upon opening a job
that has processed data and immediately calling its _stats
API only to see the model size stats are zero.
In addition, this PR refactors getting the parameters needed to
open an autodetect job:
- Previously, there was a method chaining together multiple
callbacks to the job provider.
- These methods were retrieving data via GETs which is not
going to work with index rollover.
Note, this PR is not eliminating all GETs. More work is needed
to fully support index rollover.
relates elastic/x-pack-elasticsearch#801
Original commit: elastic/x-pack-elasticsearch@1ef1d44b32
The xcontent parser was only set to read all data to a map
which did not work, when the returned data was in form of an
array (for example the cat API is doing this, if the response
format is set to JSON).
relates elastic/x-pack-elasticsearch#351
Original commit: elastic/x-pack-elasticsearch@08ad457bf6
If forced, the internal RemovePersistentTasks API is invoked instead of going through
ML. This will remove the task, which should trigger the task framework to do
necessary cleanup.
At that point, the Delete* APIs interpret a missing task as CLOSED/STOPPED,
so they can be removed regardless of the original state.
Original commit: elastic/x-pack-elasticsearch@bff23c7840
* [ML] Support all XContent types in Data API
This changes the POST Data API so that it accepts all XContent types instead of just JSON.
For now the datafeed is restricted to only sending JSON to the POST data API.
* Rename SimpleJsonRecordReader to XContentRecordReader
Also renames `DataFormat.JSON` to `DataFormat.XCONTENT`
* fixes YAML tests
Original commit: elastic/x-pack-elasticsearch@5fd20690b8
This adds basic info about jobs and datafeeds, sufficient
for the first release for supporting monitoring and
phone-home.
In particular, usage now displays:
- job count for _all and by state
- detectors statistics (min, max, mean, total) for _all jobs and by job state
- model size statistics (min, max, mean, total) for _all jobs and by job state
- datafeed count for _all and by state
Relates elastic/x-pack-elasticsearch#660
Original commit: elastic/x-pack-elasticsearch@6e0da6c3db
This change cleans up some NORELEASE comments that are either no longer relevant or actually should be TODO comments
Original commit: elastic/x-pack-elasticsearch@9947f1176e
Submit job updates to a concurrent queue when job update has been processed by ClusterService. Then from a background thread delegate the job updates to the node running the autodetect process. This maintains the same order as how the job config updates have occurred to the cluster state and thus preventing job config updates to the same job to arrive in the wrong order to the job's autodetect process. (the expectation is that in practise this will rarely happen)
The behaviour of the update api changes with this pr, because the api now returns when the update has been made to cluster state, whereas before it would return when the update was made to the autodetect process too. Updating the autodetect process happens in the background. I think that this change in behaviour is acceptable.
Use TP#scheduleWithFixedDelay(...) instead of TP#schedule(...) and
removed the custom rescheduling and cancelling.
Also changed LocalNodeMasterListener#executorName to SAME
Original commit: elastic/x-pack-elasticsearch@c24c0dd7d7
If a persistent task throws an exception, the persistent tasks framework will no longer try to restart the task. This is a temporary measure to prevent threshing the cluster with endless restart attempt. We will revisit this in the future version to make the restart process more robust. Please note, however, that if node executing the task goes down, the task will still be restarted on another node.
Original commit: elastic/x-pack-elasticsearch@30712e0fbf
* [ML] Removes direct dependency on Jackson Parser
The classes that read data from the post data action ready to write to the autodetect process had a direct dependency on Jackson’s `Parser` class. This changes makes those classes depend on XContent instead making them consistent with the way we parse requests and data across Elasticsearch and X-Pack.
* Simplify json record reader
This commit removes the unnecessary `AbstractJsonRecordReader` and `JsonRecordReader` interfaces/classes. These are not required as we do and should only have one implementation of reading json records.
Original commit: elastic/x-pack-elasticsearch@366b8af943
Removes the transport layer dependency from PersistentActions, makes PersistentActionRegistry immutable and rename actions into tasks in class and variable names.
Original commit: elastic/x-pack-elasticsearch@e3e5b79c28
The action name is currently inconsistent with the class name.
Also as more types of jobs will be added, they should reuse the existing actions,
so it make more sense to gave a generic job name for the action name. Note that
it is ok to have additional rest handlers for new type of jobs.
Original commit: elastic/x-pack-elasticsearch@a70d46fb8a
The LdapRealm submits runnables to the generic thread pool when a bind is necessary as a bind must
be a synchronous operation and we do not want to block network threads on this. However, the
generic threadpool could be full and this runnable could get queued. When this happens requests
appear to hang; to prevent this we submit a delayed executable that will stop the runnable from
attempting to connect to Ldap and provide an exceptional response to the listener.
relates elastic/x-pack-elasticsearch#716
Original commit: elastic/x-pack-elasticsearch@3e43b17f1d
This commit fixes the merging of field level security rules from multiple roles. Prior to 5.2, the
merging was treated as the merging of languages whereas after 5.2, this incorrectly became a merge
of all rules meaning a single wildcard could cause denials to be ignored.
Original commit: elastic/x-pack-elasticsearch@42f9e6d8b0
This commit removes the blocking invocation of bulk requests in monitoring as in some cases this
can lead to exhaustion of the generic threadpool, which effectively prevents the node from
operating normally.
One behavior change that is made by this commit is the MonitoringService will no longer wait
indefinitely when closing as this can lead to a node blocking forever and never shutting down.
Instead a wait of 10 seconds is added, which aligns with the security index audit trail's behavior
on shutdown.
relates elastic/x-pack-elasticsearch#715
Original commit: elastic/x-pack-elasticsearch@5ba7f49aab
In preparation of extending the model snapshot update API,
it makes sense to make it immutable to avoid any confusion
that could be caused by having setters on it.
I also like immutable stuff :-)
Original commit: elastic/x-pack-elasticsearch@6b2ee527a8
The scheduler based trigger engine is not enabled by default
as the ticker based trigger engine is used. As we dont use it
in production, this commit removes this specific implementation.
It also removes some uneeded abstractions like AbstractTriggerEngine, TriggerEngine.Listener and TriggerEngine.Job
Original commit: elastic/x-pack-elasticsearch@b17a2e9d62
This commit makes the XPackFeatureSet#usage calls asynchronous. Previously these were synchronous
calls that would execute a multi-search request from the currently elected master node in a blocking
fashion. The multi-search request is now executed asynchronously.
relates elastic/x-pack-elasticsearch#213
Original commit: elastic/x-pack-elasticsearch@a0cb988442
Datafeed state was always stopped when the stats
endpoint was called with _all. The reason is that
_all was not being expanded into all datafeed IDs.
This commit fixes the issue by expanding _all into
all datafeed IDs in the cluster. Stats are then fetched
for all of them.
relates elastic/x-pack-elasticsearch#693
Original commit: elastic/x-pack-elasticsearch@ccbdb35b6e
Prior to this change the integration tests could fail for up to 24
hours after a change to the C++ output format
Original commit: elastic/x-pack-elasticsearch@b8d9fa0adb
Rename model_debug_output to model_plot
The model debug output is 1 result type of the autodetect (anomaly detection) process reporting bounds/statistics/state - not anomalies - It's main usage as of time of writing is visualization in the UI, naming the 'blue shadow' under the result graph. Rename in order to remove 'debug' in the name
Original commit: elastic/x-pack-elasticsearch@86de428512
This commit restricts custom role providers to only apply to those
instances of x-pack with a platinum level license. All other license
types will not be allowed to use custom role providers. Any custom role
providers implemented via the XPackExtension will not take effect unless
the license is platinum.
relates elastic/x-pack-elasticsearch#720
Original commit: elastic/x-pack-elasticsearch@4fc35494ee
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.
Original commit: elastic/x-pack-elasticsearch@0b23ef528f
When adding support for rest request filtering of sensitive content, the overridden rest request
did not properly delegate the #getRemoteAddress method to the wrapped request. This resulted in a
NPE when a filtered rest request was created and an audit record needed to be generated.
relates elastic/x-pack-elasticsearch#714
Original commit: elastic/x-pack-elasticsearch@710b43355b
User signing was removed with the move to TLS only transport and this change removes a no longer
used setting.
Original commit: elastic/x-pack-elasticsearch@c221f1f4a4
* Changed ML action names to allow distinguishing of admin and read-only actions
using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
InternalClient when searching for results so that index/document level permissions
applied to ML results are respected
Original commit: elastic/x-pack-elasticsearch@eee800aaa8
This commit removes an unnecessary String creation from the char[] of a password and instead uses
a byte[] that is cleared after it is used to prevent the password bytes from sticking around in
memory longer than required.
Original commit: elastic/x-pack-elasticsearch@1154a68965
- introduce builtin user 'beats_system'
- init monitoring-beats ES mapping
- add beats to MonitoredSystem + Resolver
Original commit: elastic/x-pack-elasticsearch@4d7b45d54d
DataStreamDiagnostics
DataStreamDiagnostics analyzes input data regarding machine learning fit. It checks whether data is sane/plausible as anomaly detection on broken data (or misconfiguration).
Original commit: elastic/x-pack-elasticsearch@2f37d3c960
This commit fixes the SHA hashes for the Netty dependencies after Netty
was upgraded from version 4.1.8 to version 4.1.9.
Original commit: elastic/x-pack-elasticsearch@ca4c376886
This integration tests only checks if date math index names are
supported. This is nothing that a watcher test should check for.
Original commit: elastic/x-pack-elasticsearch@8a06988fff
Now machine learning is simply a feature of X-Pack, so I have made the
MachineLearning class more like the other feature classes. It no longer extends
Plugin and its createComponents() method takes an InternalClient rather than
a generic Client.
Original commit: elastic/x-pack-elasticsearch@704860147c
This commit adds an end-point to force deletion of expired data:
DELETE /_xpack/ml/_delete_expired_data
A few other things are changed too:
- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
visibility
- Commit results when quantiles are persisted to ensure they are visible
for renormalization
Original commit: elastic/x-pack-elasticsearch@8ca5272a94
Switched the order of the local atomic flag check and cluster state check,
based on the observation that we used to sometimes get a storm of cluster
state updates on initial startup, due to a race between checking for the
presence of an index template and starting to create one.
Original commit: elastic/x-pack-elasticsearch@7ae83648ce
* [ML] Wait for templates to be installed before running ML Integ tests
* Revert put job template check and preserve template change
* Review comments
Original commit: elastic/x-pack-elasticsearch@21deb34f4a
Security API to determine which (if any) of a specified set of index/cluster privileges are held by the current (runAs) user.
Intended for use by Kibana to distinguish between read/write and read-only users, but should be applicable to other uses cases also.
Closes: elastic/x-pack-elasticsearch#282
Original commit: elastic/x-pack-elasticsearch@8b4cfdb858
The only tricky thing here was what to do about ES jars. We now ignore them for the jar sha checking, and the base elasticsearch license and notice is copied here, and elasticsearch jars are mapped to those license/notice files.
Original commit: elastic/x-pack-elasticsearch@a6373cfe4e
This fixes returning results only for the job asked
when a shared index is used.
The commit also refactors the field count check to
use the field mappings API and solves a bug where the
check blows due to the _default_ type not having properties.
[Zach] Minor test tweak to MlJobIT
Original commit: elastic/x-pack-elasticsearch@729f886879
The yaml test runner now throws error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which was previously ignored. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.
Original commit: elastic/x-pack-elasticsearch@920201207c
Only distributed ml tests, only enable ml as security, watcher, monitoring etc. only add noise to the tests.
Disable mock big arrays for distributed ml tests, as nodes starting/stopping in tests somehow leaves allocations around,
which only is a test problem.
Original commit: elastic/x-pack-elasticsearch@5ff1e69036
This moves the index structure to using a single, shared index
(.ml-anomalies-shared). Custom indices can still be used by manually
setting `results_index`.
An alias is always created which points from `.ml-anomalies-<jobid>`
to `.ml-anomalies-shared`.
User defined indices are prepended with "custom-"
Index helper functions have been renamed to make this clear. Furthermore,
accessing an index should always be done either by fetching the
currently configured index/alias from the state, or using the preconfigured
alias. Because the user can specify a custom physical index, it is
impossible to determine the physical index "by convention" now.
The helpers have been configured to reflect that.
Original commit: elastic/x-pack-elasticsearch@a5368eb230
This commit adds the ability for x-pack extensions to optionally
provide custom roles providers, which are used to resolve any roles
into role descriptors that are not found in the reserved or native
realms. This feature enables the ability to define and provide roles
from other sources, without having to pre-define such roles in the security
config files.
relates elastic/x-pack-elasticsearch#77
Original commit: elastic/x-pack-elasticsearch@bbbe7a49bf
As part of authentication, we use a iterating action listener to perform asynchronous authentication against the realm
chain. When this listener is called with a response or a failure, it could be called from a thread that is not owned by
the Elasticsearch threadpool such as a LDAPConnectionReader thread. When this happens, we need to ensure that the
ThreadContext is not left with items in it otherwise we leave behind things like Authentication and hit obscure errors.
This commit stores the context when the listener calls the consumer or onResponse/onFailure is invoked, which prevents
us from polluting a external thread's ThreadContext.
Original commit: elastic/x-pack-elasticsearch@0f50fb6c10
Some json builders in the codebase were not closed. even
though this is not needed for the BytesStreamOutput being used,
there is more closing logic in the jackson classes, which we
should not rely on, that those never change or are ok to not
close.
Original commit: elastic/x-pack-elasticsearch@05a43d80ff
This is the Elastic convention, and also makes it clearer where the
actual log message from the C++ starts
Original commit: elastic/x-pack-elasticsearch@dc9aeefb08
in case of tribe node client do not even bother to check for native process version, because it isn't running.
Original commit: elastic/x-pack-elasticsearch@e6fdd5e82a
[ML] * Add MachineLearningTemplateRegistry class
[ML] * Add blocking method to put templates required by tests
[ML] * Add version check for templates
[ML] * Review comments
Original commit: elastic/x-pack-elasticsearch@07d315e56d
This norelease can be removed, because we no longer create indices automatically when missing, but use index templates.
Original commit: elastic/x-pack-elasticsearch@b6880ce7b7
Also included assignment explanation to both job and datafeed stats apis and
included executor node to datafeed stats api
Original commit: elastic/x-pack-elasticsearch@783bc77ef6
When the datafeed indexes are concrete indexes, the search
will throw IndexNotFoundException, thus a notification is being
generated. However, when the indexes are patterns, the search
does not fail.
This commit adds a warning when a lookback-only datafeed
retrieves no data as it is certain that the user misconfigured
something and it's useful to make him aware of it.
Original commit: elastic/x-pack-elasticsearch@e3490ea978
If any format other than json is sent, it should get converted to json, as watcher works under the assumption that the script is a template, hence json.
Original commit: elastic/x-pack-elasticsearch@554bcd306b
Fixeselastic/prelert-legacy#803
On Windows the JVM hangs during shutdown if the native controller process
is still running. A thread dump suggests this is because of a finalizer being
blocked. We can avoid the problem by explicitly telling the controller to exit
when the node is stopped. Although the hang is Windows-specific, it doesn't
hurt to do this on *nix too.
Original commit: elastic/x-pack-elasticsearch@6fa1c2b966
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners
This fix was mistakenly removed in another commit
Original commit: elastic/x-pack-elasticsearch@6305456d5a
Adds a preview end-point: `/_xpack/ml/datafeeds/{datafeed_id}/_preview`
The endpoint returns an array with the JSON records that will reach
writer to the c++ process. Thus, the preview can be used to verify
that a datafeed is configured correctly with regard to itself and
its corresponding job.
Original commit: elastic/x-pack-elasticsearch@1e0e9c906d
* [ML] Index template for notifications, meta and state indices
* Add ignore_unavailable option to searches so missing indices don’t throw errors
* Test for job existence before returning results
* Template mapping for results indices
* Fix tests
* Reinstate merge conflict dropped code
* Fix and tidy up JobProviderTests
Tests had badly named & duplicated tests
* Don’t check if the job indices already exist when allocating
Indices are created on demand from templates
* No need to create indices in initialisation service
Using templates now
* Remove unused code and add comment
Original commit: elastic/x-pack-elasticsearch@1ba115d4fc
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners
Original commit: elastic/x-pack-elasticsearch@a7aff45571
Refactors xcontent serialization of Request and Status to use their writable names instead of action name. That simplifies the parsing logic, allows reuse of the same status object for multiple actions and is consistent with how named objects in xcontent are used.
Original commit: elastic/x-pack-elasticsearch@3d059ea952
The DnRoleMapper was changed when moving to asynchronous authentication, which introduced a regression that leads to a
NPE being thrown on file refresh. The cause is the addition of a null listener from the LdapRealm upon construction.
This change removes the listener constructor argument as it was always null or a empty runnable and adds a check in the
addListener method to ensure the listener is not null. Finally, the DnRoleMapperTests had a bug where the temporary
filenames did not have the correct suffix and tests were not actually testing what was intended.
relates elastic/x-pack-elasticsearch#608
Original commit: elastic/x-pack-elasticsearch@f47f258590
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.
This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.
This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.
Original commit: elastic/x-pack-elasticsearch@b31d144597
In rare cases, the CertificateToolTests#testPromptingForInstanceInformation could try to use the same name multiple
times. This causes the test to fail with a message for an unknown prompt. This commit ensures no duplicates are added.
relates elastic/x-pack-elasticsearch#597
Original commit: elastic/x-pack-elasticsearch@ab8bb7ad50
This commit allows persistent actions to indicate why a task was or wasn't assigned to a certain node.
Original commit: elastic/x-pack-elasticsearch@cdacf9b10f
Changed `ProblemTracker#updateEmptyDataCount(boolean)` into `ProblemTracker#reportEmptyDataCount()`.
and added `ProblemTracker#reportNoneEmptyCount()` to reset the empty count counter.
Original commit: elastic/x-pack-elasticsearch@c1a44d6fd3
This allows it to be disabled/enabled via job updates.
It also simplifies it by removing bounds_percentile as it currently
rarely makes sense to set the bounds to another value than 95.0.
Original commit: elastic/x-pack-elasticsearch@c27fce2d86
This PR introduces a service component that is scheduled to run
once per day, around 30 minutes past midnight, local time.
Listeners can be subscribed to the service in order to get notified
when the next service trigger occurs so that they can execute
management tasks.
Currently, there are 2 such listeners: one is performing removal
of results and the other of model snapshots that have expired
their retention period.
Finally, note that the service is only installed on the master
node to ensure there is only one running on the cluster.
Original commit: elastic/x-pack-elasticsearch@c8edb6016a
Let LicensingTests.testNativeRealmMigratorWorksUnderBasicLicense wait for 3s (instead of 1s) as it is occasionally failing.
Also added logging so that if/when it does fail we can see how long we were waiting and what was taking place.
Original commit: elastic/x-pack-elasticsearch@22882ec412
Added missing named xcontent entries, tweaked start datafeed validation logic and fixed status serialization issue that were uncovered by this test.
Original commit: elastic/x-pack-elasticsearch@8b3fb012df
* Wrap exceptions if they are not ElasticsearchExceptions in TransportJobTaskAction
* Review comments
Original commit: elastic/x-pack-elasticsearch@07e900ffaa
Remove `node.attr.xpack.ml.allocation_enabled` node attribute and instead use `node.attr.max_running_jobs` attribute to indicate a node is a ml node.
(in addition to indicating how many jobs can hosted on a node)
Original commit: elastic/x-pack-elasticsearch@7979bc55b4
Fields were missing from the job's copy constructor.
This was the cause of intermittent test failures.
Original commit: elastic/x-pack-elasticsearch@3cd705ab3c
...correctly update the job's model_snapshot_id whenever
a new model snapshot is persisted. Load the snapshot that
corresponds to the job's model_snapshot_id when a job is
opened.
This PR makes a series of changes that completes the change
of having job's active model snapshot persisted on the job.
Original commit: elastic/x-pack-elasticsearch@c2e23fa1ee
A job allocation is either a job task in OPENING state or a job task that has been assigned to an executor node, but the executor node hasn't had the opportunity to set the job task status to OPENING.
In order to keep track of restarted tasks, `allocationIdOnLastStatusUpdate` field was added to `PersistentTaskInProgress` class.
This will allow persistent task implementors to detect whether the executor node has changed or has been unset since the last status update has occured.
Original commit: elastic/x-pack-elasticsearch@b7b85a8274
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.
Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.
Original commit: elastic/x-pack-elasticsearch@29d336a008
All validation should be done in the builder and the Job object should just be a dumb, immutable blob.
Original commit: elastic/x-pack-elasticsearch@a78e6edbe9
* Update running process from cluster state update
* Put a big lock on the UpdateProcessAction
* Protect concurrent update of the process with a semaphore
* Review comments
Original commit: elastic/x-pack-elasticsearch@d08b53247a
This commit moves persistent tasks from ClusterState.Custom to MetaData.Custom and adds ability for the task to remain in the metadata after completion.
Original commit: elastic/x-pack-elasticsearch@751b40de2a
The messages stored in the .ml-notifications index include a
message field that is mapped to text. It makes sense to keep
that mapping as it can improve search experience in messages.
However, the ML kibana app needs to aggregate over messages,
thus we also need to provide a keyword mapping.
Original commit: elastic/x-pack-elasticsearch@5a5df53d85
* Update input field mappings if job results index exists
* Add field names to the index mapping in shared indices
* Throw error if creating job will violate the index.mapping.total_fields.limit
* Review comments
Original commit: elastic/x-pack-elasticsearch@d7712d9263
This commit ensures that the authentication sets the correct version on the stream when it is serialized over the wire
so that there is not a version mismatch between the authentication and the connection it came from.
Original commit: elastic/x-pack-elasticsearch@267d7068f4
Any failure to communicate with an LDAP server should be treated as if authentication failed rather than a fatal "stop the world" error.
Because the AD realm validates the user's password by attempting to bind to the directory with that username/password combination, connection failures are to be expected and must be handled gracefully because the AD realm may not be the last realm in the order-chain, and any exception would prevent later realms from being tested.
Closes: elastic/x-pack-elasticsearch#564
Original commit: elastic/x-pack-elasticsearch@5282290b00
This change adds checks to the following machine learning actions to check that machine learning is permitted by the license before executing the request and throw and exception if the license forbids it:
* Create job
* Create data feed
* Open job
* Start data feed
This is not a final list of the restrictions we want to have in place if the license forbids ML access but it serves as a starting point and can be easily updated following further discussions.
The change also moves the `transportClientMode` check to `createComponents` so we don’t try to start the server parts of machine learning (job manager, connection to named pipes etc.) if we are running in a transport client.
Original commit: elastic/x-pack-elasticsearch@6c19ebd3bc
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.
Original commit: elastic/x-pack-elasticsearch@89e4875f6c
Now that TLS is required for node to node communication, we no longer need to sign the
authentication information to prevent tampering.
Original commit: elastic/x-pack-elasticsearch@1f86cf2395
Was accidentally injecting the script object, not the string version of the code. Also
added a check so we only inject for Painless scripts (and not groovy, etc).
Minor style tweaks too.
Original commit: elastic/x-pack-elasticsearch@58c7275bd8
Essentially an update to https://github.com/elastic/prelert-legacy/pull/736 . Still does not
default to using shared indices, but adds the capability for two jobs to share the same one
without conflict
Still does not default to using shared indices, just adds the capability for two jobs to share the same one without conflict.
Original commit: elastic/x-pack-elasticsearch@60d93a06ea
By injecting immediately before use, we don't have to store it in the config and return
huge blobs to the user when they request the config
Original commit: elastic/x-pack-elasticsearch@2e53c930ea
Source filtering is currently written as if the fact that an object is accepted
means that all sub fields and objects are implicitly accepted, which is not
true.
Original commit: elastic/x-pack-elasticsearch@afc4440e7a
This node attribute setting defaults to true when ml is enabled and jobs are only opened on nodes where the allocation enabled node attribute is set to true.
Original commit: elastic/x-pack-elasticsearch@fd3f1b3058
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.
Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`
Original commit: elastic/x-pack-elasticsearch@48ea9d781b
This commit removes the undocumented, test used only, setting xpack.monitoring.collection.collectors. It should not be used by users at all, and has been initially created for collector tests.
From elastic/elasticsearch#4957
Original commit: elastic/x-pack-elasticsearch@0fa39a94b5
The Elastic Secrets vault is served via HTTPS with a Let's Encrypt
certificate. The root certificate that cross-signed the Let's Encrypt
certificates were not trusted by the JDK until 8u101. This commit adds a
version check at the start of the build to make it clear the cause of
the underlying issue, and what the fix is.
Relates elastic/x-pack-elasticsearch#541
Original commit: elastic/x-pack-elasticsearch@6bf8076cb6
Dimitrios Athanasiou
jaymode
David Roberts
Tim Vernum
Ryan Ernst
David Kyle
Hendrik Muhs
Tanguy Leroux
Jay Modi
Chris Earle
Nik Everett
Tim Vernum
Igor Motov
Adrien Grand
Alexander Reelsen
Jim Ferenczi
Luca Cavanna
Tim Brooks
Daniel Mitterdorfer
Jason Tedor
Yannick Welsch
Zachary Tong
Martijn van Groningen
Colin Goodheart-Smithe
Chris Earle
Ali Beyad
Simon Willnauer
Zachary Tong
javanna
Robin Clarke
Boaz Leskes
Steffen Siering
Zachary Tong