This removes the "node" type from `.monitoring-data-2`. This data is sent to _both_ the time-based and non-time-based indexes for Elasticsearch, but the UI only used the time-based variant already.
This is another step in the process of removing the `.monitoring-data-2` index. There is now only one `_type` left in that index: `cluster_info`, which is used by the UI and phone home stats because it contains the license details _and_ the `stack_stats` (e.g., `xpack_usage`).
Original commit: elastic/x-pack-elasticsearch@2cadb5939d
This is a task towards allowing rollover.
Multiple model_size_stats are stored in order to allow
analytics of the memory usage over time. The job _stats
need to display the latest model_size_stats. Before this
commit, the latest model_size_stats was being stored with
a special ID and it was retrieved using that ID. This
does not lend itself well for rollover as we would end up
with multiple of those special IDs in the rolled indices.
This commit removes the need to store a special model_size_stats
version. Job _stats now retrieve the model_size_stats by searching
for the latest one. It also uses a manual ID for all model_size_stats
in order to maintain a single document per log_time.
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@b2796e9b08
Now that the Monitoring UI no longer checks the `.monitoring-data-2` index
for Kibana, Logstash, or Beats data, we can stop accepting the duplicated
data in that index (the _exact_ same documents are also indexed into the
time-based index for each product).
This ignores rather than rejects requests that contain such documents to
allow older clients to communicate with a 5.5+ monitoring cluster.
Original commit: elastic/x-pack-elasticsearch@def472cf2e
- role_mapping.native is always present, but contains no entries if the security index is unavailable
- file based role mapping does not allow duplicate keys
Original commit: elastic/x-pack-elasticsearch@734cf7e2c0
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.
When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.
Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.
Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
In some cases (based on the randomisation) the primary-realm would have no mappings, but the secondary-realm would.
If this occurred we wouldn't write out any mappings and the secondary-realm would not authorize the access being tested
Original commit: elastic/x-pack-elasticsearch@8e81ec9dd7
This introduces a role-mapping API to X-Pack security.
Features:
- A `GET`/`PUT`/`DELETE` API at `/_xpack/security/role_mapping/`
- Role-mappings are stored in the `.security` index
- A custom expression language (in JSON) for expressing the mapping rules
- Supported in LDAP/AD and PKI realms
- LDAP realm also supports loading arbitrary meta-data (which can be used in the mapping rules)
- A CompositeRoleMapper unifies roles from the existing file based mapper, and the new API based mapper.
- Usage stats for native role mappings
Original commit: elastic/x-pack-elasticsearch@d9972ed1da
This is related to elastic/elasticsearch#24412. That commit changed how
ListenableActionFuture implementations are created. This commit
updates x-pack to be compatible with those changes. In particular, all
the usages of ListenableActionFuture in x-pack could be replaced with
PlainActionFuture as the "listening" functionality was not being used.
Original commit: elastic/x-pack-elasticsearch@7c8d8e3df9
JobDataDeleter handles the deletion logic for 3 cases:
1. deleting a model snapshot and its state docs
2. deleting all results after a timestamp
3. deleting all interim results
The last 2 are currently implemented by manually performing
a search and scroll and then adding matching hits in a bulk
delete action. This operation is exactly what delete-by-query
does.
This commit changes JobDataDeleter to use delete-by-query. This
makes the code simpler and less error-prone. The downside is
losing some logging which seems non-critical. Unit tests for
JobDataDeleter are also removed as they are heavily mocked tests,
adding little value and high maintenance cost. This functionality
is tested by integration tests already.
relates elastic/x-pack-elasticsearch#821
Original commit: elastic/x-pack-elasticsearch@7da91332bd
The distribution of watches now happens on the node which holds the
watches index, instead of on the master node. This requires several
changes to the current implementation.
1. Running on shards and replicas
In order to run watches on the nodes with the watches index on its
primaries and replicas. To ensure that watches do not run twice, there is
a logic which checks the local shards, runs a murmurhash on the id and
runs modulo against the number of shards and replicas, this is the way to
find out, if a watch should run local. Reloading happens
2. Several master node actions moved to a HandledTransportAction, as they
are basically just aliases for indexing actions, among them the
put/delete/get watch actions, the acknowledgement action, the de/activate
actions
3. Stats action moved to a broadcast node action, because we potentially
have to query every node to get watcher statistics
4. Starting/Stopping watcher now is a master node action, which updates
the cluster state and then listeners acts on those. Because of this watches
can be running on two systems, if you those have different cluster state
versions, until the new watcher state is propagated
5. Watcher is started on all nodes now. With the exception of the ticker
schedule engine most classes do not need a lot of resources while running.
However they have to run, because of the execute watch API, which can hit
any node - it does not make sense to find the right shard for this watch
and only then execute (as this also has to work with a watch, that has not
been stored before)
6. By using a indexing operation listener, each storing of a watch now
parses the watch first and only stores on successful parsing
7. Execute watch API now uses the watcher threadpool for execution
8. Getting the number of watches for the stats now simply queries the
different execution engines, how many watches are scheduled, so this is
not doing a search anymore
There will be follow up commits on this one, mainly to ensure BWC compatibility.
Original commit: elastic/x-pack-elasticsearch@0adb46e658
Within the same JVM, setting the number of processors available to Netty
can only be done once. However, tests randomize the number of processors
and so without intervention would attempt to set this value multiple
times. Therefore, we need to use a flag that prevents setting this value
in tests.
Relates elastic/x-pack-elasticsearch#1266
Original commit: elastic/x-pack-elasticsearch@d127149725
Many of the tests assume that the trial license has already been generated before the test gets to run. As this is asynchronously triggered upon node
startup, however, there is no guarantee that trial license generation has completed before the tests get to execute, leading to null values when
checking clusterService.state().metaData().custom(LicensesMetaData.TYPE).
Original commit: elastic/x-pack-elasticsearch@d909c9ba95
LicenseManagerServiceTests sometimes fails in jenkins, but fairly
rarely. We don't get useful logs when it does. This cranks up
the log level and adds some more assertions so we can better track
down where the failure comes from.
Relates to elastic/x-pack-elasticsearch#222
Original commit: elastic/x-pack-elasticsearch@3e08725fc7
When a user creates a datafeed, as well as checking they have permission
to create a datafeed we also check that they have permission to search the
indices they've configured the datafeed to search.
Previously this second check was erroneously done for the user who issued
the put_datafeed request, whereas it should be done as the runas user for
that request.
Original commit: elastic/x-pack-elasticsearch@4c35204c66
When we revert to snapshot, if we delete intervening results
we should delete with an open end on the time range for the
case when future data has been posted to the job.
Original commit: elastic/x-pack-elasticsearch@c3f5e8f19e
The one argument ctor for `Script` creates a script with the
default language but most usages of are for testing and either
don't care about the language or are for use with
`MockScriptEngine`. This replaces most usages of the one argument
ctor on `Script` with calls to `ESTestCase#mockScript` to make
it clear that the tests don't need the default scripting language.
Original commit: elastic/x-pack-elasticsearch@c1d05b7357
When the active directory realm was refactored to add support for authenticating against multiple
domains, only the default authenticator respected the user_search.filter setting. This commit moves
this down to the base authenticator and also changes the UPN filter to not include sAMAccountName
in the filter.
Original commit: elastic/x-pack-elasticsearch@d2c19c9bee
- Removes need to handle exception from action methods
- Clearly renames DatafeedJobIT to DatafeedJobsRestIT to distinguish
from DatafeedJobsIT
- Refactors DatafeedJobsIT to reuse MlNativeAutodetectIntegTestCase
Original commit: elastic/x-pack-elasticsearch@5bd0c01391
Cross cluster search uses ClusterSearchShardsAction under the covers.
Without this change, you would need both "read_cross_cluster" and "view_index_metadata" privilegs in order to have permission to execute searches from a remote cluster.
Original commit: elastic/x-pack-elasticsearch@65a6aff329
If a single permission set does not have a query defined then this should be considered as the user
not having document level security for the indices matching that pattern. However, the lack of
document level security was not being taken into account and document level security was being
applied when it should not have been.
Original commit: elastic/x-pack-elasticsearch@f5777c2f37
Many tests in monitoring use the pattern of calling first awaitMonitoringDocsCount, and then doing a search that checks certain properties, assuming
that the doc count is correct at that point. In the presence of replicas, awaitMonitoringDocsCount might not wait for all shard copies to have the
desired property. A subsequent search might then hit a shard where the property does not hold. As these tests randomize the number of replicas
(through the random_index_template), it easier to constrain awaitMonitoringDocsCount just to the primary and then do subsequent checks just by
querying the primary.
Original commit: elastic/x-pack-elasticsearch@4165beb903
This commit creates a single server socket that will be connected to by local sockets. The local
sockets will use the port of the previously stopped ldap server as the local port. This will
prevent the ldap library from establishing a connection. The previous use of server sockets for
this did not work on all operating systems as the backlog parameter has platform specific meaning.
Original commit: elastic/x-pack-elasticsearch@03b6bf39d4
This commit adds a timeout for the expiration of invalidated tokens so that we can expect that the
requests will have been finished before we do the assertions on the internal test cluster.
Original commit: elastic/x-pack-elasticsearch@2928706224
This commit adds a token based access mechanism that is a subset of the OAuth 2.0 protocol. The
token mechanism takes the same values as a OAuth 2 standard (defined in RFC 6749 and RFC 6750),
but differs in that we use XContent for the body instead of form encoded values. Additionally, this
PR provides a mechanism for expiration of a token; this can be used to implement logout
functionality that prevents the token from being used again.
The actual tokens are encrypted using AES-GCM, which also provides authentication. The key for
encryption is derived from a salt value and a passphrase that is stored on each node in the
secure settings store. By default, the tokens have an expiration time of 20 minutes and is
configurable up to a maximum of one hour.
Relates elastic/x-pack-elasticsearch#8
Original commit: elastic/x-pack-elasticsearch@3d201ac2bf
This is an issue where a bucket can have both interim results and
non-interim results, a bucket should never have both at the same time.
The steps to cause this situation are:
1. Flush a running job and create interim results
2. Close that job (this does not delete interim results)
3. Re-open the job and POST data
4. The job will eventually emit a bucket result which mingles with the
existing interim results
Normally interim results are deleted by AutoDetectResultProcessor when a
bucket is parsed following a flush command. Because of the close and
re-opening of the job AutoDetectResultProcessor no longer knows that a
previous flush command creating interim results.
The fix is to always delete interim results the first time
AutoDetectResultProcessor sees a bucket.
relates elastic/x-pack-elasticsearch#1188
Original commit: elastic/x-pack-elasticsearch@5326455f54
In the SessionFactoryLoadBalancingTests, we sometime want a connection to a certain IP and Port to
fail as a way to mock an unresponsive/disconnected LDAP server. The test does this by starting up
multiple LDAP servers and then shutting some down. When the server is shut down the port that it
was bound to is open for another process or test to bind to, which can lead to sporadic failures in
CI. This change is a best effort attempt to prevent this by binding a server socket to the port and
filling its backlog so other connections should fail.
Relates elastic/x-pack-elasticsearch#1195
Original commit: elastic/x-pack-elasticsearch@b31a560c93
The DatafeedJobsIT.testRealtime test fails from time to time.
The test seems to take a long time to execute the flush action
after the lookback. This could make sense as the test produces
a few records over the span of a week with 5 minutes bucket_span.
Thus, flush will end up doing a lot of word to create results
for so many buckets.
This change increases the bucket_span to 1 hour. Hopefully, this
will stop the failures.
Relates elastic/x-pack-elasticsearch#1162
Original commit: elastic/x-pack-elasticsearch@4366907371
This commit fixes the support for elliptic curve certificates that are specified as a PEM file.
These certificates and private keys can now be read properly and a integration test was added to
ensure that TLS also functions correctly with these certificates.
Original commit: elastic/x-pack-elasticsearch@6d6d579c88
This commit reduces spamming of the logs when a common SSL exception is encountered such as a
client not trusting the server's certificate or a plaintext request sent to a channel that expects
TLS traffic.
relates elastic/x-pack-elasticsearch#1062
Original commit: elastic/x-pack-elasticsearch@94959e79f6
This change prevents the situation where cleanup of ML indices immediately
after deleting a job leaves the audit notification in limbo because the index
it was due to be indexed into has been deleted.
Relates elastic/x-pack-elasticsearch#1142
Original commit: elastic/x-pack-elasticsearch@300e9c36ce
Ordinary Kibana users should not have access to the cluster state of ES,
and therefore they should not be able to access ML jobs without explicit
permission.
Original commit: elastic/x-pack-elasticsearch@77273d561a
When a condition is unmet, the ack status of the actions needs to be
resetted again, so that new alerts can be triggered.
Due to a bugfix this functionality was removed from ES 5.0.0-alpha5
onwards.
relates elastic/x-pack-elasticsearch#1123
Original commit: elastic/x-pack-elasticsearch@83db2cecf9
Persistent tasks should verify that completion notification is done for correct version of the task, otherwise a delayed notification from an old node can accidentally close a newly reassigned task.
Original commit: elastic/x-pack-elasticsearch@478bb6e730
* Adds a check to wait for active tasks for XPackRestIT
* uses test logger
* Change to use assertBusy instead of awaitBusy
* fixes failures with active tasks remaining
* Moves wait for pending tasks into MlRestTestStateCleaner
* remove unecessary log line
Original commit: elastic/x-pack-elasticsearch@1f098dbb64
By creating the watches via the exporter, we get to afford ourselves
with a much more automatic and simpler set of security permissions.
This does limit us in a few ways (e.g., every exporter has to deal with
cluster alerts itself, which means that newer releases of Kibana cannot
help by adding newer cluster alerts for older, still-monitored
clusters).
Original commit: elastic/x-pack-elasticsearch@448ef313c3
When Logstash 5.2 - 5.3 submit documents via the `_xpack/monitoring/_bulk`
endpoint, it sends its time-based documents with an explicit `_id` of
`""`.
This used to be automatically ignored by Monitoring, but we now accept the
_id that we are given (including `null`). ES, prior to 5.3.1, accepted
`""` as a valid `_id` through the `_bulk` endpoint, which means that it
blindly accepted and overwrote documents given that ID, meaning that all
Logstash instances "shared" the exact same document and therefore the UI
becomes useless.
This change allows `""` to be used and it simply replaces that value, and
only that value, with `null`. This enables backwards compatibility with LS
5.2 - 5.3.0.
Original commit: elastic/x-pack-elasticsearch@889578e61e
PersistentTasksCustomMetadata was using a generic param named `Params`. This conflicted with the imported interface `ToXContent.Params`. The java compiler was preferring the generic param over the interface so everything was fine but Eclipse apparently prefers the interface int his case which was screwing up the Hierarchy and causing compile errors in Eclipse. This changes fixes it by renaming the Generic param to `P`
Original commit: elastic/x-pack-elasticsearch@8528870684
- Mark all security indices (that is all indices managed by SecurityLifecycleService) as "superuser only" (only superuser role can have direct permissions)
- Add unit tests for IndexLifecycleManager
Original commit: elastic/x-pack-elasticsearch@e4478825e0
This commit removes the SecuredString class that was previously used throughout the security code
and replaces it with the SecureString class from core that was added as part of the new secure
settings infrastructure.
relates elastic/x-pack-elasticsearch#421
Original commit: elastic/x-pack-elasticsearch@e9cd117ca1
When a index name pattern contains both date math and wildcards, the name resolution does not
return the expected result. This change moves the date math resolution to before our attempts to
match wildcards so that both can be used in the same pattern.
relates elastic/x-pack-elasticsearch#1065
Original commit: elastic/x-pack-elasticsearch@9f48b42fad
let close job and stop datafeed apis redirect to elected master node.
This is for cluster state observation purposes, so that a subsequent open and then close job or
start and then stop datafeed see the same local cluster state and sanity validation doesn't fail.
Original commit: elastic/x-pack-elasticsearch@21a63184b9
introduced separate task names to register the persistent tasks executors and params.
Also renamed start and stop datafeed action names to be singular in order to be consistent with open and close action names.
Original commit: elastic/x-pack-elasticsearch@21f7b242cf
Support for default settings has been removed in core and so some
methods were refactored. This commit responds to this change in core.
Original commit: elastic/x-pack-elasticsearch@b22c612de4
The path has changed so it’s no longer possible to distinguish between data feed and job tasks.
The preceding test get_datafeed provides ample coverage anyway.
Original commit: elastic/x-pack-elasticsearch@780b1beb6b
When the execute watch API is called without recording the execution
in the watch history, the watch status is not updated, in order to not
divert the in-memory object status and the one persisted on disk.
In order to work around this issue, the execute watch API can simply
clone a new watch status and a new watch, which means the object in
the watch store is never updated. This allows for execution and changing
of the watch status, before it is returned to the client.
relates elastic/x-pack-elasticsearch#889
Original commit: elastic/x-pack-elasticsearch@6a0d9c9a78
Changes persistent task serialization and forces params and status to have the same writeable name as the task itself.
Original commit: elastic/x-pack-elasticsearch@59cf3dca39
remove `node.attr.max_running_jobs` node attribute and use `node.attr.ml.enabled` node attribute instead to know whether a node is a ml node or not.
Also renamed `max_running_jobs` setting to `xpack.ml.max_running_jobs`.
Original commit: elastic/x-pack-elasticsearch@798732886b
Removes the last pieces of ActionRequest from PersistentTaskRequest and renames it into PersistTaskParams, which is now just an interface that extends NamedWriteable and ToXContent.
Original commit: elastic/x-pack-elasticsearch@5a298b924f
Following this change, if the user runs on a platform that we don't ship
ML binaries for:
* If ML is enabled the node still refuses to start, but clearly says why
* If ML is disabled the node starts up without logging any errors
Original commit: elastic/x-pack-elasticsearch@af4fb8c411
Now that task id are strings instead of longs (elastic/x-pack-elasticsearch#1035), ml can use the job and datafeed as task id.
This removes logic that would otherwise iterate over all tasks and check if the task's request id was equal to the provided id and instead just do lookup in the task map.
Job and datafeed task ids are prefixed with either 'job-' or 'datafeed-', because job and datafeed ids don't have to be unique as they are stored separately from each other.
Original commit: elastic/x-pack-elasticsearch@b48c2b368a
This built-in watcher_admin role is able to execute all watcher actions,
read the watch history indices and read the watches index
index. The watcher_user role allows to GET a watch and to get the stats and thats it.
relates elastic/x-pack-elasticsearch#978
Original commit: elastic/x-pack-elasticsearch@11b33a413b
- stops the datafeed when post/flush throw a conflict exception.
A conflict exception signifies the job state is not opened, thus
we are better off stopping the datafeed.
- handles flushing the job the same way as posting to the job.
relates elastic/x-pack-elasticsearch#855
Original commit: elastic/x-pack-elasticsearch@49a54912c2
Makes the log more readable in editors not set to UTF-8.
Customers may well be in this situation on Linux/Windows.
Original commit: elastic/x-pack-elasticsearch@4e59fc90cf
The commit changes how LocalExporterTests stops: it now uses the
node_stats document collected on each node and check if it's older
than a given number of seconds (10). It also removes log traces.
Original commit: elastic/x-pack-elasticsearch@0384690b41
Before this change the persistent task operations related to opening
and closing jobs would time out a long time before the operations
related to native processes.
Original commit: elastic/x-pack-elasticsearch@23076b773b
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"
Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request.
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.
This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
Closes: elastic/x-pack-elasticsearch#887
Original commit: elastic/x-pack-elasticsearch@b81b363729
The PR detects if SMILE is being provided, then correctly slices the stream such that each document is parsed individually. This is required because jackson's SMILE parser is stricter than it's JSON parser and will stop parsing when it hits a streamSeparator (unlike JSON, which will eagerly try to find more objects to parse).
Removes the forced-headers from the various REST tests.
relates elastic/x-pack-elasticsearch#642
Original commit: elastic/x-pack-elasticsearch@c0e97cd545
Instead of having a separate listener for indicating that the current task is finished, this commit is switching to use allocated object itself.
Original commit: elastic/x-pack-elasticsearch@7ad5362121
`PersistentTasksExecutor#getAssignment(...)` should be a cheap and side-effect free method,
but in case of `OpenJobPersistentTasksExecutor` and `StartDatafeedPersistentTasksExecutor` before this change it would index a document each time `getAssignment(...)` was invoked
Original commit: elastic/x-pack-elasticsearch@5ca5890baf
The change applies chunking by default on aggregated datafeeds.
The chunking is set to a manual mode with time_span being
1000 histogram buckets.
The motivation for the change is two-fold:
1. It helps to avoid memory pressure/blowing.
Users may perform a lookback on a very long period of time. In that
case, we may hold a search response for all that time which could
include too many buckets. By chunking, we avoid that situation
as we know we'll only keep results for 1000 buckets at a time.
2. It makes cancellation more responsive.
In elastic/x-pack-elasticsearch#862 we made the processing of a search response cancellable in a
responsive manner. However, the search phase cannot be cancelled at
the moment. Chunking makes the search phase shorter, which will
result to a better user experience when they stop an aggregated
datafeed.
Also note the change sets the default chunking_config on datafeed
creation so the setting is no longer hidden.
Relates to elastic/x-pack-elasticsearch#803
Original commit: elastic/x-pack-elasticsearch@ae8f120f5f
When a datafeed task is created but it cannot be assigned the task
has a null status. This means _stats report it as stopped, however
deleting it fails. In addition, it's a better experience to error
the start datafeed request all together and give the user the chance
to fix his data indices.
This change fails a datafeed-start if it cannot be assigned.
relates elastic/x-pack-elasticsearch#1018
Original commit: elastic/x-pack-elasticsearch@532288fda0
Retries should be already handled by TransportMasterNodeAction, there is no need to introduce another retry layer in Persistent Tasks code.
Original commit: elastic/x-pack-elasticsearch@967ac7f7fa
This commit changes how LocalExporterTests stops the monitoring
components: it first stops the monitoring service (but keeps the
local exporter enabled), deletes and checks if monitoring indices
are recreated, and then disables the local exporter.
Original commit: elastic/x-pack-elasticsearch@4c4809a660
Closing a job may take a while. In the meantime it is possible to start a datafeed, because before this change the job state remained OPENED.
With this change when the executor node receives the close job request, it will first set the status to CLOSING and after that closes the job (closing autodetect process, etc.).
relates elastic/x-pack-elasticsearch#990
Original commit: elastic/x-pack-elasticsearch@d8d89c0756
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.
Original commit: elastic/x-pack-elasticsearch@f1665815c2
The execution has diverged too much from post data, flush and update process apis, since the close all jobs have been added.
The logic is now easier to understand as it exist in a single source file instead of in both CloseJobAction and TransportJobTaskAction.
Original commit: elastic/x-pack-elasticsearch@daf5fabad5
Users currently have difficulty diagnosing authentication failures.
Some logging messages mislead them, and in other cases there are unexpected behaviours that are not logged at all.
These additional DEBUG log messages and change some existing messages in an attempt to alleviate that problem.
Original commit: elastic/x-pack-elasticsearch@c6ea98b038
Chris Earle
Tanguy Leroux
Dimitris Athanasiou
Tim Vernum
Igor Motov
Adrien Grand
Alexander Reelsen
Jim Ferenczi
Tim Vernum
Luca Cavanna
Tim Brooks
Hendrik Muhs
Daniel Mitterdorfer
David Kyle
Jason Tedor
Yannick Welsch
Nik Everett
David Roberts
Zachary Tong
Jay Modi
jaymode
Martijn van Groningen
Ryan Ernst
Colin Goodheart-Smithe
Chris Earle
Ali Beyad
Simon Willnauer
Zachary Tong
javanna