This is mainly a commit to reduce noise in test logfiles when going
through them. When watcher shuts down and another node takes over, it
might try to start watcher again and tries to load triggered watches.
However the triggered watches index could be gone in the meantime due to
further shutdown. This results in logging a stack trace that the index
does not exist.
This commit checks the cluster state before trying to load triggered
watches to prevent an IndexNotFoundException in the logs.
Original commit: elastic/x-pack-elasticsearch@9f26d557d0
This introduces a check to only pause the execution of watcher
when there is no metadata but there was a shard on this node before
that inside of the ClusterStateListener.
This prevents repeated logging that watcher was paused even
though it was not necessary to call anything.
Original commit: elastic/x-pack-elasticsearch@8d3a829ffb
Watcher had an undocumented feature to configure the settings of any
index template via updating the cluster settings. Instead of changing
the template one could add a setting during runtime that overwrote
what is written in the index template.
As index template are created once and not overwritten, users should
just change the index template - or the concrete index settings like
number of replicas.
This feature was not exposed in our documentation at all.
Original commit: elastic/x-pack-elasticsearch@32e1769925
We wait for index templates to be installed before running tests, but
these can get upgraded when the master node is upgraded. If we don't
wait again in this case then tests can fail due to the cleanup code
being overwhelmed by pending cluster state updates that weren't waited
for before the test.
Original commit: elastic/x-pack-elasticsearch@eff0b24c11
This removes the Cluster State collector and resolver and moves the collection of the cluster state (and cluster health, which is already included in cluster stats).
This makes the tests a little more stable and removes an extra network hop during monitoring data collection.
Original commit: elastic/x-pack-elasticsearch@44851d2dd6
Reworks the rolling restart tests so that all configuration
options share code. Now there is a project per configuration:
* `qa:rolling-upgrade:with-ssl-with-system-key`
* `qa:rolling-upgrade:with-ssl-without-system-key`
Original commit: elastic/x-pack-elasticsearch@5672b4a4f2
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
Original commit: elastic/x-pack-elasticsearch@b07aa78a7b
The aim of this change is to prevent many identical requests to create
watcher index templates being submitted when a cluster first starts up
and many cluster state updates are happening. Prior to this change, if
watcher's original index template creation requests queued up behind other
cluster state change requests then for each other request watcher would
re-request creation of all its index templates. After this change it
uses a strategy similar to that used by ML to only have one creation
request per index template in the cluster state change queue at any time.
Relates elastic/x-pack-elasticsearch#1368
Relates elastic/x-pack-elasticsearch#1631
Relates elastic/x-pack-elasticsearch#1650
Original commit: elastic/x-pack-elasticsearch@ad87bf3f78
Although the job stats for jobs with missing results indices are clearly
ruined, it's better to provide zeroes for the missing values and show the
stats for other jobs than to fail the whole request. This means the UI
can continue to function.
relates elastic/x-pack-elasticsearch#1656
Original commit: elastic/x-pack-elasticsearch@a06fa994a5
* Wait for job deletion if it is in the deleting state
* Tolerate errors if multiple force delete requests
Original commit: elastic/x-pack-elasticsearch@1f0c9fbb86
When a user executes a watch and specifies it as part of the
execute watch API, no triggered watch is created, as the watch
cannot be picked up anymore (it only leaves for the duration of
the request).
However until now the TriggeredWatchStore was invoked and tried
to delete this non-existing triggered watch, resulting in some
log cluttering.
This commit removes this try to delete a non-existing triggered
watch.
Original commit: elastic/x-pack-elasticsearch@3db125cea2
This is just the culmination of all of the minor PRs associated with 1068. It will:
- Drop the `.monitoring-data-N` index
- Drop use of `_type` in all cases (replaced by `doc` and a new `type` field)
- Drop the API version from the template name (e.g., instead of `.monitoring-es-6` we now use `.monitoring-es`).
- Change API version to `-6-` from `-2-`.
- Both exporters handle versioned resources (templates, pipelines, and watches)
- HTTP exporters will optionally (true by default) publish placeholders for the old, `-2` templates.
When this is backported, it will need to:
- Change `index_patterns` to `template` within the templates.
- Downgrade the version requirements for the templates, pipeline, and watches _and_ the HTTP exporter itself (all require 6.0)
This is a companion to the feature branch in X-Pack Kibana elastic/x-pack-kibana/pull/1318 and they need to be merged at the same time.
Original commit: elastic/x-pack-elasticsearch@6031cfffa4
This commit adds better security for scroll requests in that they are now tied to a single user as
we only authorize the request that creates the scroll. This is accomplished by adding a
SearchOperationListener that listens for new scroll contexts and stores the authentication on the
ScrollContext. Then upon
retrieval of the search context for a query or fetch, the current authentication is compared to the
authentication that was present when the scroll context was created. If the current authentication
belongs to a different user, then a SearchContextMissingException will be thrown to prevent leaking
a valid vs invalid scroll id.
Additionally, signing of a scroll id is only performed when there is a older node in the cluster
that would expect the scroll id to be signed. Once this is backported to 5.x, we can remove this
bwc layer for 6.0/master.
Original commit: elastic/x-pack-elasticsearch@0e5dcafd32
When a user or client intend to delete a datafeed
and its job, there is benefit into ensuring the
datafeed has gracefully stopped (ie no data loss).
In constrast, the desired behaviour is to stop and
delete the datafeed as quickly as possible.
This change adds a force option to the delete
datafeed action. When the delete is forced,
the datafeed is isolated, its task removed and,
finally, the datafeed itself is removed from the
metadata.
relates elastic/x-pack-elasticsearch#1533
Original commit: elastic/x-pack-elasticsearch@5ae0168bf2
We try to install empty ML metadata as soon as possible after startup
if none exists. However, this still leaves a short gap when the cluster
is active with no ML metadata. To avoid problems, functions that use
the ML metadata should treat this situation as equivalent to having
empty ML metadata.
relates elastic/x-pack-elasticsearch#1643
Original commit: elastic/x-pack-elasticsearch@8f0e00cda8
ignoreAliases allows to resolve index expressions against concrete indices only, rather than against indices and aliases. It is used for now only in IndicesAliasesRequest and the indices resolution code in the security plugin needs to be adapted accordingly.
Original commit: elastic/x-pack-elasticsearch@ae964eade9
This commit switches over to two index aliases per job: one for reading
and one for writing. In the future this will allow the addition of a
rollover endpoint for ML results indices. (Rollover is still not possible
following this change, but the change to make it possible in the future
should not be a breaking change now.)
Relates elastic/x-pack-elasticsearch#1599
relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@d648f4631f
* Add force delete job option
* Can’t kill a process on a 5.4 node
* Address review comments
* Rename KillAutodetectAction -> KillProcessAction
* Review comments
* Cancelling task is superfluous after it has been killed
* Update docs
* Revert "Cancelling task is superfluous after it has been killed"
This reverts commit 576950e2e1ee095b38174d8b71de353c082ae953.
* Remove unnecessary TODOs and logic that doesn't alwasys force close
Original commit: elastic/x-pack-elasticsearch@f8c8b38217
This test would sometime leak threads.
The "Timer thread for LDAPConnection" is created by the unboundid SDK - closing the connection should force the thread to terminate
Original commit: elastic/x-pack-elasticsearch@bd58a17a59
The cluster state listener to decide if watcher should be reloaded was
assuming that no aliases could be used and thus wrongly could trigger
a reload, which could have lead to wrong test results.
During debugging I also added a reason for reloading and fixed another
wrong test assumption.
Also the listener does not rely on previous cluster state, but stores this
in instance variable, as we need to compare with local state and not the
previous cluster state.
Original commit: elastic/x-pack-elasticsearch@582783a66d
Includes:
- Extensive changes to "mapping roles" section
- New section for role mapping API
- Updates to LDAP/AD/PKI realms to refer to API based role mapping
- Updates to LDAP/AD realms: `unmapped_groups_as_roles` only looks at file-based mappings
- Updates to LDAP/AD realms: new setting for "metadata"
Original commit: elastic/x-pack-elasticsearch@6349f665f5
* [DOCS] Clarify ML node settings re transport requests
* [DOCS] Clarify xpack.ml.enabled based on feedback
Original commit: elastic/x-pack-elasticsearch@3102d1e3f3
Alexander Reelsen
Ryan Ernst
Lisa Cawley
David Roberts
David Kyle
Chris Earle
Chris Earle
Nik Everett
jaymode
Tim Vernum
Jack Conradson
Jason Tedor
Igor Motov
Jay Modi
Martijn van Groningen
Dimitris Athanasiou
Luca Cavanna