PersistentTasksCustomMetadata was using a generic param named `Params`. This conflicted with the imported interface `ToXContent.Params`. The java compiler was preferring the generic param over the interface so everything was fine but Eclipse apparently prefers the interface int his case which was screwing up the Hierarchy and causing compile errors in Eclipse. This changes fixes it by renaming the Generic param to `P`
Original commit: elastic/x-pack-elasticsearch@8528870684
- Mark all security indices (that is all indices managed by SecurityLifecycleService) as "superuser only" (only superuser role can have direct permissions)
- Add unit tests for IndexLifecycleManager
Original commit: elastic/x-pack-elasticsearch@e4478825e0
This commit removes the SecuredString class that was previously used throughout the security code
and replaces it with the SecureString class from core that was added as part of the new secure
settings infrastructure.
relates elastic/x-pack-elasticsearch#421
Original commit: elastic/x-pack-elasticsearch@e9cd117ca1
When a index name pattern contains both date math and wildcards, the name resolution does not
return the expected result. This change moves the date math resolution to before our attempts to
match wildcards so that both can be used in the same pattern.
relates elastic/x-pack-elasticsearch#1065
Original commit: elastic/x-pack-elasticsearch@9f48b42fad
let close job and stop datafeed apis redirect to elected master node.
This is for cluster state observation purposes, so that a subsequent open and then close job or
start and then stop datafeed see the same local cluster state and sanity validation doesn't fail.
Original commit: elastic/x-pack-elasticsearch@21a63184b9
introduced separate task names to register the persistent tasks executors and params.
Also renamed start and stop datafeed action names to be singular in order to be consistent with open and close action names.
Original commit: elastic/x-pack-elasticsearch@21f7b242cf
Support for default settings has been removed in core and so some
methods were refactored. This commit responds to this change in core.
Original commit: elastic/x-pack-elasticsearch@b22c612de4
The path has changed so it’s no longer possible to distinguish between data feed and job tasks.
The preceding test get_datafeed provides ample coverage anyway.
Original commit: elastic/x-pack-elasticsearch@780b1beb6b
When the execute watch API is called without recording the execution
in the watch history, the watch status is not updated, in order to not
divert the in-memory object status and the one persisted on disk.
In order to work around this issue, the execute watch API can simply
clone a new watch status and a new watch, which means the object in
the watch store is never updated. This allows for execution and changing
of the watch status, before it is returned to the client.
relates elastic/x-pack-elasticsearch#889
Original commit: elastic/x-pack-elasticsearch@6a0d9c9a78
Changes persistent task serialization and forces params and status to have the same writeable name as the task itself.
Original commit: elastic/x-pack-elasticsearch@59cf3dca39
remove `node.attr.max_running_jobs` node attribute and use `node.attr.ml.enabled` node attribute instead to know whether a node is a ml node or not.
Also renamed `max_running_jobs` setting to `xpack.ml.max_running_jobs`.
Original commit: elastic/x-pack-elasticsearch@798732886b
Removes the last pieces of ActionRequest from PersistentTaskRequest and renames it into PersistTaskParams, which is now just an interface that extends NamedWriteable and ToXContent.
Original commit: elastic/x-pack-elasticsearch@5a298b924f
Following this change, if the user runs on a platform that we don't ship
ML binaries for:
* If ML is enabled the node still refuses to start, but clearly says why
* If ML is disabled the node starts up without logging any errors
Original commit: elastic/x-pack-elasticsearch@af4fb8c411
Now that task id are strings instead of longs (elastic/x-pack-elasticsearch#1035), ml can use the job and datafeed as task id.
This removes logic that would otherwise iterate over all tasks and check if the task's request id was equal to the provided id and instead just do lookup in the task map.
Job and datafeed task ids are prefixed with either 'job-' or 'datafeed-', because job and datafeed ids don't have to be unique as they are stored separately from each other.
Original commit: elastic/x-pack-elasticsearch@b48c2b368a
This built-in watcher_admin role is able to execute all watcher actions,
read the watch history indices and read the watches index
index. The watcher_user role allows to GET a watch and to get the stats and thats it.
relates elastic/x-pack-elasticsearch#978
Original commit: elastic/x-pack-elasticsearch@11b33a413b
- stops the datafeed when post/flush throw a conflict exception.
A conflict exception signifies the job state is not opened, thus
we are better off stopping the datafeed.
- handles flushing the job the same way as posting to the job.
relates elastic/x-pack-elasticsearch#855
Original commit: elastic/x-pack-elasticsearch@49a54912c2
Makes the log more readable in editors not set to UTF-8.
Customers may well be in this situation on Linux/Windows.
Original commit: elastic/x-pack-elasticsearch@4e59fc90cf
The commit changes how LocalExporterTests stops: it now uses the
node_stats document collected on each node and check if it's older
than a given number of seconds (10). It also removes log traces.
Original commit: elastic/x-pack-elasticsearch@0384690b41
Before this change the persistent task operations related to opening
and closing jobs would time out a long time before the operations
related to native processes.
Original commit: elastic/x-pack-elasticsearch@23076b773b
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"
Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request.
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.
This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
Closes: elastic/x-pack-elasticsearch#887
Original commit: elastic/x-pack-elasticsearch@b81b363729
The PR detects if SMILE is being provided, then correctly slices the stream such that each document is parsed individually. This is required because jackson's SMILE parser is stricter than it's JSON parser and will stop parsing when it hits a streamSeparator (unlike JSON, which will eagerly try to find more objects to parse).
Removes the forced-headers from the various REST tests.
relates elastic/x-pack-elasticsearch#642
Original commit: elastic/x-pack-elasticsearch@c0e97cd545
Instead of having a separate listener for indicating that the current task is finished, this commit is switching to use allocated object itself.
Original commit: elastic/x-pack-elasticsearch@7ad5362121
`PersistentTasksExecutor#getAssignment(...)` should be a cheap and side-effect free method,
but in case of `OpenJobPersistentTasksExecutor` and `StartDatafeedPersistentTasksExecutor` before this change it would index a document each time `getAssignment(...)` was invoked
Original commit: elastic/x-pack-elasticsearch@5ca5890baf
The change applies chunking by default on aggregated datafeeds.
The chunking is set to a manual mode with time_span being
1000 histogram buckets.
The motivation for the change is two-fold:
1. It helps to avoid memory pressure/blowing.
Users may perform a lookback on a very long period of time. In that
case, we may hold a search response for all that time which could
include too many buckets. By chunking, we avoid that situation
as we know we'll only keep results for 1000 buckets at a time.
2. It makes cancellation more responsive.
In elastic/x-pack-elasticsearch#862 we made the processing of a search response cancellable in a
responsive manner. However, the search phase cannot be cancelled at
the moment. Chunking makes the search phase shorter, which will
result to a better user experience when they stop an aggregated
datafeed.
Also note the change sets the default chunking_config on datafeed
creation so the setting is no longer hidden.
Relates to elastic/x-pack-elasticsearch#803
Original commit: elastic/x-pack-elasticsearch@ae8f120f5f
When a datafeed task is created but it cannot be assigned the task
has a null status. This means _stats report it as stopped, however
deleting it fails. In addition, it's a better experience to error
the start datafeed request all together and give the user the chance
to fix his data indices.
This change fails a datafeed-start if it cannot be assigned.
relates elastic/x-pack-elasticsearch#1018
Original commit: elastic/x-pack-elasticsearch@532288fda0
Retries should be already handled by TransportMasterNodeAction, there is no need to introduce another retry layer in Persistent Tasks code.
Original commit: elastic/x-pack-elasticsearch@967ac7f7fa
This commit changes how LocalExporterTests stops the monitoring
components: it first stops the monitoring service (but keeps the
local exporter enabled), deletes and checks if monitoring indices
are recreated, and then disables the local exporter.
Original commit: elastic/x-pack-elasticsearch@4c4809a660
Closing a job may take a while. In the meantime it is possible to start a datafeed, because before this change the job state remained OPENED.
With this change when the executor node receives the close job request, it will first set the status to CLOSING and after that closes the job (closing autodetect process, etc.).
relates elastic/x-pack-elasticsearch#990
Original commit: elastic/x-pack-elasticsearch@d8d89c0756
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.
Original commit: elastic/x-pack-elasticsearch@f1665815c2
The execution has diverged too much from post data, flush and update process apis, since the close all jobs have been added.
The logic is now easier to understand as it exist in a single source file instead of in both CloseJobAction and TransportJobTaskAction.
Original commit: elastic/x-pack-elasticsearch@daf5fabad5
Users currently have difficulty diagnosing authentication failures.
Some logging messages mislead them, and in other cases there are unexpected behaviours that are not logged at all.
These additional DEBUG log messages and change some existing messages in an attempt to alleviate that problem.
Original commit: elastic/x-pack-elasticsearch@c6ea98b038
Increase the timeout to give enough time for a datafeed to
stop smoothly.
This is the second step to avoid hitting the default timeout.
The first was ensuring aggregated datafeed is cancellable in
a responsive manner. The third and final step will be to
apply chunking in aggregated datafeeds in order to shorten
the duration of the search, which will make cancellation even
more responsive.
Relates elastic/x-pack-elasticsearch#803
Original commit: elastic/x-pack-elasticsearch@db642330ec
This commit restores the ability to build x-pack-elasticsearch without issues when running without
access to the internet. When the `--offline` flag is used, we will not try to contact vault and the
aws apis to retrieve the ml-cpp binaries but instead gradle will use a cached version even though
it may be expired.
relates elastic/x-pack-elasticsearch#726
Original commit: elastic/x-pack-elasticsearch@b0915d8fa9
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.
relates elastic/x-pack-elasticsearch#870
Original commit: elastic/x-pack-elasticsearch@34bdce6e99
This commit renames and moves the forked delete by query classes from being ml specific to being a
xpack common class since an upcoming security feature plans to make use of this. Additionally, this
commit fixes a issue where the dbq action was being executed by the calling user instead of the
xpack user for certain requests. This was found when adding a authorization change that restricts
this action's execution to the xpack user only.
Original commit: elastic/x-pack-elasticsearch@d5967e7255
There was a problem with the way CompositeBytesReference was used in the
StateProcessor. In the case of a large state document we ended up with a
deeply nested CompositeBytesReference that then caused a deep stack and N^2
processing in the bulk action processor.
This change uses an intermediate list of byte arrays that get combined into
a single CompositeBytesReference to avoid the deep nesting.
Additionally, errors in state processing now bubble up to close the state
stream, which will cause the C++ process to stop trying to persist more state.
Finally, the results processor also times out after a similar period (30 minutes)
to that used by the state processor.
Original commit: elastic/x-pack-elasticsearch@ceb31481d1
Rather than using an async call, this leverages
the Assignment logic while selecting nodes.
Now with 300% more tests!
Original commit: elastic/x-pack-elasticsearch@300d628f72
Colin Goodheart-Smithe
Tim Vernum
Jay Modi
jaymode
Martijn van Groningen
Jason Tedor
Dimitrios Athanasiou
Alexander Reelsen
Ali Beyad
David Kyle
Igor Motov
David Roberts
Zachary Tong
Tanguy Leroux
Hendrik Muhs
Simon Willnauer
Zachary Tong
javanna
Luca Cavanna
Ryan Ernst
Adrien Grand