Every cluster state update resulted in a log message, that watcher
pauses execution. This has been fixed to only log, if there was an
actual state switch from executing to pausing, but do nothing if
there are no local shards anyway.
This will reduce the logging noise in tests a lot.
Original commit: elastic/x-pack-elasticsearch@32ab86610c
* Give kill a chance to kill the process before closing input
* Remove variable that can be refactored out
Original commit: elastic/x-pack-elasticsearch@42f7a3cece
The graph API needs to be able to search in remote indices. Although it uses the Search API to perform the search and so doesn’t need to deal with remote indexes directly, the security feature needs to know it can be used with remote indexes so it knows to include remote indices in the list of indices accessible from the API for index level security
Original commit: elastic/x-pack-elasticsearch@e3cd84963e
This change removes all local security checks against remote cluster names.
Any user is allowed to attempt a cross-cluster search, and it is the responsibility of the remote cluster to authorise the search (or not).
This includes support for remote searches even if you have _no_ local search privileges.
Original commit: elastic/x-pack-elasticsearch@1620c3a8fa
Currently, aggregated datafeeds construct JSON from the aggregation
response by traversing all nested aggregations. In order to
achieve this, multiple leaf aggregations are not supported. Also,
scenarios it makes it impossible to effectively use pipeline
aggregations as it will not ignore the intermediate bucket
aggregations.
This commit refactors AggregationToJsonProcessor in order to
support the above scenarios. This is achieved by only converting
the fields of interest, that is the job analysis fields.
Original commit: elastic/x-pack-elasticsearch@8b575956ca
This changes the validation criteria we use for user and role
names in the file realm, native realm, and the
realm-agnostic code in x-pack security. The new criteria is:
A valid username's length must be at least 1 and no more than 1024
characters. It may not contain leading or trailing whitespace. All
characters in the name must be be alphanumeric (`a-z`, `A-Z`, `0-9`),
printable punctuation or symbols in the https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block)[Basic Latin (ASCII) block],
or the space character.
Original commit: elastic/x-pack-elasticsearch@f77640f269
Today we have some hidden complexity related to default configurations
might specify NO_KEY which is in some cases valid for server configuration.
This change removes the leniencey paramenters on the validation methods and removes
obsolet asserts.
Original commit: elastic/x-pack-elasticsearch@17ed4b1d20
You can now disable cluster alerts in the local exporter, which you can do in the HTTP exporter already.
This helps users that mess up their watcher configuration (e.g., disabling scripts) can turn off the feature to avoid log spam.
Original commit: elastic/x-pack-elasticsearch@f2096b553d
The cluster alert did not get updated when we dropped the logical 'type' for cluster_state in favor of merging it with cluster_stats in 5.5+.
Original commit: elastic/x-pack-elasticsearch@c7105be36f
Today we parse / construct SSLConfiguration late when client / server
channels are created. This is problematic if we try to read from secure settings
in the future since we need to read all secure settings as soon as the node is
constructed. If we keep on reading late, we will access a closed keystore
since channel creation happens during node startup.
Original commit: elastic/x-pack-elasticsearch@78d6061990
If the internal index version of an index is not the right one, do
not start watcher.
Also, add the internal index version of 6 to all our index templates.
Original commit: elastic/x-pack-elasticsearch@20b50aa82b
If multiple job deletion requests were sent in quick succession, there was a
race condition that meant they could both get through the check to enforce
one active deletion request at a time. Then, if the job was immediately
recreated after the first deletion request returned, the second, still running,
deletion request could interfere with it and delete the aliases that the put
job request created.
This problem can be avoided by using the "ask forgiveness, not permission"
idiom when checking if the job is already being deleted at the beginning of
each deletion request.
Additionally, now even force delete requests will wait for a certain amount
of time for a prior delete request to complete. This is to avoid the same
race conditions. However, force delete requests will eventually start an
(unsafe) parallel delete to provide a get-out in case a delete request
completely dies.
relates elastic/x-pack-elasticsearch#1765
Original commit: elastic/x-pack-elasticsearch@b5c8f26a0e
The current testing setup only checked if watcher was started, but it
also needs to check for the index template in order to be sure that
everything is set up correctly, before trying to put a watch.
relates elastic/x-pack-elasticsearch#1762
Original commit: elastic/x-pack-elasticsearch@3ed78b15a1
This commit changes a couple of places where our ExceptionsHelper
class was throwing exceptions to instead return the exceptions.
Then they can be passed to onFailure() methods or thrown depending
on what's appropriate for the caller. This is the standard Elastic
way of handling failures.
Original commit: elastic/x-pack-elasticsearch@fce07eb075
This allows to configure a proxy for the reporting attachment
action. The proxy is used by the HTTP client.
Original commit: elastic/x-pack-elasticsearch@87b6ab1b68
The MockTerminal used in tests uses \n always and the reverted commit re-introduced the bug which
had been fixed earlier.
Original commit: elastic/x-pack-elasticsearch@09b93b5565
This change enables closing a job while it is in
the middle of restoring its state. This is has the
benefit of allowing users to close jobs that due to
relocation are `opened` but they are still restoring
state. It also helps avoiding race conditions in tests.
Part of this change also includes restoring the state
as a separate step from the process creation. This means
we no longer block the job map while the process is
restoring its state.
relates elastic/x-pack-elasticsearch#1270
Original commit: elastic/x-pack-elasticsearch@1713a4a7c4
In cases where the job is bound on the analytics performance
the datafeed can fail because the scroll expires. This is
commit increases the scroll context duration from 10 to 30 minutes
as a temporary solution that will avoid most cases.
Original commit: elastic/x-pack-elasticsearch@fd277bbaa1
This changes the native realm migrate tool tests to use the System.lineSeperator instead of `\n`
so that the tests will pass on Windows.
Original commit: elastic/x-pack-elasticsearch@d3f9a71ac4
Prior to this change, if the persistent tasks framework noticed that a
job was running on a node that was isolated but has rejoined the cluster
then it would close that job. This was not ideal, because then the job
would persist state from the autodetect process that was isolated. This
commit changes the behaviour to kill the autodetect process associated
with such a job, so that it does not interfere with the autodetect process
that is running on the node where the persistent tasks framework thinks it
should be running.
In order to achieve this a change has also been made to the behaviour of
force-close. Previously this would result in the autodetect process being
gracefully shut down asynchronously to the force-close request. However,
the mechanism by which this happened was the same as the mechanism for
cancelling tasks that end up running on more than one node due to nodes
becoming isolated from the cluster. Therefore, force-close now also kills
the autodetect process rather than gracefully stopping it. The documentation
has been changed to reflect this. It should not be a problem as force-close
is supposed to be a last resort for when normal close fails.
relates elastic/x-pack-elasticsearch#1186
Original commit: elastic/x-pack-elasticsearch@578c944371
This came up in a forum post. An NPE was raised, when a search input
contained a search that did not contain a body, but just specified
indices or types.
This commit allows for empty bodies, and also makes sure there are
no null pointer exceptions by using empty bytes references otherwise.
In addition a suite scoped integration test was converted to a unit
test.
Original commit: elastic/x-pack-elasticsearch@29be2976fc
This introduces a new index setting called xpack.internal.format to
x-pack, which is configured for all of our index templates and set to
"v6". This indicates the version of compatibility of this index.
In addition a setting named index.xpack.version has been removed,
as it was unused.
Watcher does not start, if the watches and the triggered watches
index is not compatible with this setting.
Original commit: elastic/x-pack-elasticsearch@e430691c51
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
Original commit: elastic/x-pack-elasticsearch@d01b0df1d9
Before the event stats were mislabeled, so were not being indexed, and
the ephemeral_id was only in the _state document.
Original commit: elastic/x-pack-elasticsearch@ca0ec81aa5
Adds tests similar to `:qa:full-cluster-restart` for x-pack. You
run them with `gradle :x-pack:qa:full-cluster-restart:check`.
The actual tests are as basic as it gets: create a doc and load it,
shut down, upgrade to master, startup, and load it. Create a user
and load it, shut down, upgrade to master, startup, and load it.
Relates to elastic/x-pack-elasticsearch#1629
Original commit: elastic/x-pack-elasticsearch@8994bec8e7
This commit removes unnecessary initialization of the system key in tests that no longer make use
of the system key. It also removes the feature usage for the system key in the SecurityFeatureSet.
Original commit: elastic/x-pack-elasticsearch@b9fffe0bd3
Logstash now has ephemeral id at the instance level and also at the
pipeline level, we need to add them to the logstash monitoring template.
Original commit: elastic/x-pack-elasticsearch@dfac702d59
* Use bulk request to persist model plots and model size stats
* Revert persisting model size stats in the bulk request
* Refactor results persister
Original commit: elastic/x-pack-elasticsearch@f51297bfc2
We recently added logging for critical authentication failures as they had previously been silent (with respect to logs) but would cause authentication processing to stop.
However, the reserved realm intentionally uses exceptions to stop any other realm authenticating a reserved user if the password is entered incorrectly.
Since this is the most common use of exceptions in the authc chain, we reduce the logging verbosity in normal cases (drop the stack trace, remove "unexpected") and only log the full details in debug.
Original commit: elastic/x-pack-elasticsearch@686a98010b
This is related to elastic/x-pack-elasticsearch#1217. This change introduces a tool
bin/x-pack/setup-passwords that will streamline the setting of
internal user passwords. There are two modes of operation. One mode
called auto, automatically generates passwords and prints them to
the console. The second mode called interactive allows the user to
enter passwords.
All passwords are changed using the elastic superuser. The elastic
password is the first password to be set.
Original commit: elastic/x-pack-elasticsearch@00974234a2
After improving the authorization of scroll requests and backporting to 5.x, we no longer need to
have any signing code in master. This commit removes it.
Original commit: elastic/x-pack-elasticsearch@8b65fd9338
If an exception occurs while sending the initial setup messages to the autodetect
such that it fails rather than reaching the open state then the autodetect process
needs to be killed to prevent it hogging resources.
Relates elastic/x-pack-elasticsearch#1684
Original commit: elastic/x-pack-elasticsearch@1ee80ed9b0
In does not make sense for the time_field in the data_description to
be used as a by/over/partition field name, nor the summary_count_field,
categorization_field or as an influencer. Therefore, configurations
where the time_field in the data_description is used in the
analysis_config are now rejected.
Additionally, it causes a problem communicating with the C++ code if
the control field name (which is '.') is used in the analysis_config,
so this is also rejected at the validation stage.
Relates elastic/x-pack-elasticsearch#1684
Original commit: elastic/x-pack-elasticsearch@e6750a2cda
- Don't attempt to upgrade from 2.x
- Attempt up to 10 retries if the migration fails (with increasing back-off between attempts)
- If a cached user is disabled, recheck with the underlying store
The last change is required if the migration takes a long time.
While users are being migrated, they might be marked as disabled, but when the migration is complete they need to be usable immediately.
Original commit: elastic/x-pack-elasticsearch@2621867014
Removes the `assemble` task from projects that aren't published
to speed up `gradle assemble` so the unified release can call it.
Original commit: elastic/x-pack-elasticsearch@43dfcc15f3
These tests are starting their own nodes and do not use the testing
trigger schedule class.
There are occasional test failure due to a race condition where watcher
is in the process of being started, but cannot be shut down properly,
because starting up was not finished when the shut down was called for.
These filter tests do not rely on watcher, so we can disable them for
now, but we still need to fix a race condition in starting/stopping
watcher.
relates elastic/x-pack-elasticsearch#1422
Original commit: elastic/x-pack-elasticsearch@f13bb7a6fb
There is no need to handle any _status field in
the 6.0 release from now on, as everything has been
taken care in the upgrade API.
Original commit: elastic/x-pack-elasticsearch@606581f4d7
This changes part of the logic that was added in elastic/x-pack-elasticsearch#644 and extended
in elastic/x-pack-elasticsearch#1495 so that when ML is disabled we never try to communicate with
the native controller during node shutdown.
The original reason for needing to communicate with the native controller
when ML is disabled was the problem of elastic/prelert-legacy#803.
However, this was fixed in a better way in elastic/elasticsearch#24579.
Now there is considerable benefit in never talking to the native
controller from the plugin code when ML is disabled, because it means
anyone suffering some obscure problem with ML can disable it without
running the risk of uncovering some other obscure problem with shutdown.
Original commit: elastic/x-pack-elasticsearch@9d329483a7
This adds a check in the REST tests to ensure that
watcher is started, and if not, tries to start watcher.
This eliminates test failures where watcher was not in
the correct state due to other tests stopping watcher.
Original commit: elastic/x-pack-elasticsearch@fc547d49b4
This commit adds a new Logstash component to x-pack to support the config management work. Currently, the functionality in this component is really simple; all it does is upload a new index template for `.logstash` index. This index stores the actual LS configuration.
On this template is bootstrapped in ES, Kibana can write user-created LS configs which adhere to the mapping defined here. In the future, we're looking into adding more functionality on the ES side to handle config documents, but for now, this is simple.
relates elastic/x-pack-elasticsearch#1499, relates elastic/x-pack-elasticsearch#1471
Original commit: elastic/x-pack-elasticsearch@d7cc8675f7
In the case where a field is a text multi-field, it has
no doc values and it is not in source. Thus, the datafeed
will not be able to extract it.
However, it is possible to extract it by getting its parent
field instead. This commit implements the logic to look
in parent fields when the field in question is a text field.
Original commit: elastic/x-pack-elasticsearch@f116e89921
In 5.4.x, the datafeed attempts to get all fields from
doc_values by default. It has a `_source` parameter which
when enabled changes the strategy to instead try to get
all fields from the source.
This has been the most common issue users have been
reporting as it means the datafeed will fail to fetch
any text fields by default.
This change uses the field capabilities API in order
to automatically detect whether a field is aggregatable.
It then extracts such fields from doc_values while the
rest are taken from source. The change also adds
validation to the start datafeed action so that if
fields are missing mappings or the time field is not
aggregatable we respond with an appropriate error.
relates elastic/x-pack-elasticsearch#1649
Original commit: elastic/x-pack-elasticsearch@76e2cc6cb2
Specifying types for a datafeed should be optional
as no types is equal to searching through all types.
Original commit: elastic/x-pack-elasticsearch@f61ac01b45
Only log an entry when an actual stop is executed instead of
always logging.
Also added a reason to stop watcher to the methods, so that
debug logs will yield that information.
Original commit: elastic/x-pack-elasticsearch@8efaed0e9a
The constructor for CreateIndexResponse changed to include the index
name. This commit adapts x-pack-elasticsearch to this change.
Original commit: elastic/x-pack-elasticsearch@b078d80cd9
Commit elastic/x-pack-elasticsearch@b07aa78a7b was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement and project name.
Original commit: elastic/x-pack-elasticsearch@9a77269fa6
This is mainly a commit to reduce noise in test logfiles when going
through them. When watcher shuts down and another node takes over, it
might try to start watcher again and tries to load triggered watches.
However the triggered watches index could be gone in the meantime due to
further shutdown. This results in logging a stack trace that the index
does not exist.
This commit checks the cluster state before trying to load triggered
watches to prevent an IndexNotFoundException in the logs.
Original commit: elastic/x-pack-elasticsearch@9f26d557d0
This introduces a check to only pause the execution of watcher
when there is no metadata but there was a shard on this node before
that inside of the ClusterStateListener.
This prevents repeated logging that watcher was paused even
though it was not necessary to call anything.
Original commit: elastic/x-pack-elasticsearch@8d3a829ffb
Watcher had an undocumented feature to configure the settings of any
index template via updating the cluster settings. Instead of changing
the template one could add a setting during runtime that overwrote
what is written in the index template.
As index template are created once and not overwritten, users should
just change the index template - or the concrete index settings like
number of replicas.
This feature was not exposed in our documentation at all.
Original commit: elastic/x-pack-elasticsearch@32e1769925
This removes the Cluster State collector and resolver and moves the collection of the cluster state (and cluster health, which is already included in cluster stats).
This makes the tests a little more stable and removes an extra network hop during monitoring data collection.
Original commit: elastic/x-pack-elasticsearch@44851d2dd6
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
Original commit: elastic/x-pack-elasticsearch@b07aa78a7b
The aim of this change is to prevent many identical requests to create
watcher index templates being submitted when a cluster first starts up
and many cluster state updates are happening. Prior to this change, if
watcher's original index template creation requests queued up behind other
cluster state change requests then for each other request watcher would
re-request creation of all its index templates. After this change it
uses a strategy similar to that used by ML to only have one creation
request per index template in the cluster state change queue at any time.
Relates elastic/x-pack-elasticsearch#1368
Relates elastic/x-pack-elasticsearch#1631
Relates elastic/x-pack-elasticsearch#1650
Original commit: elastic/x-pack-elasticsearch@ad87bf3f78
David Kyle
Dimitris Athanasiou
Alexander Reelsen
Colin Goodheart-Smithe
Tim Vernum
Andy Bristol
Simon Willnauer
Adrien Grand
Chris Earle
David Roberts
Jay Modi
Martijn van Groningen
jaymode
Shaunak Kashyap
Luca Cavanna
Nik Everett
Andrew Cholakian
Pier-Hugues Pellerin
Tim Brooks
Jim Ferenczi
Boaz Leskes
Igor Motov
Ryan Ernst
Suyog Rao
Jason Tedor
Chris Earle