In 5.4.x, the datafeed attempts to get all fields from
doc_values by default. It has a `_source` parameter which
when enabled changes the strategy to instead try to get
all fields from the source.
This has been the most common issue users have been
reporting as it means the datafeed will fail to fetch
any text fields by default.
This change uses the field capabilities API in order
to automatically detect whether a field is aggregatable.
It then extracts such fields from doc_values while the
rest are taken from source. The change also adds
validation to the start datafeed action so that if
fields are missing mappings or the time field is not
aggregatable we respond with an appropriate error.
relates elastic/x-pack-elasticsearch#1649
Original commit: elastic/x-pack-elasticsearch@76e2cc6cb2
Specifying types for a datafeed should be optional
as no types is equal to searching through all types.
Original commit: elastic/x-pack-elasticsearch@f61ac01b45
Only log an entry when an actual stop is executed instead of
always logging.
Also added a reason to stop watcher to the methods, so that
debug logs will yield that information.
Original commit: elastic/x-pack-elasticsearch@8efaed0e9a
The constructor for CreateIndexResponse changed to include the index
name. This commit adapts x-pack-elasticsearch to this change.
Original commit: elastic/x-pack-elasticsearch@b078d80cd9
Commit elastic/x-pack-elasticsearch@b07aa78a7b was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement and project name.
Original commit: elastic/x-pack-elasticsearch@9a77269fa6
This is mainly a commit to reduce noise in test logfiles when going
through them. When watcher shuts down and another node takes over, it
might try to start watcher again and tries to load triggered watches.
However the triggered watches index could be gone in the meantime due to
further shutdown. This results in logging a stack trace that the index
does not exist.
This commit checks the cluster state before trying to load triggered
watches to prevent an IndexNotFoundException in the logs.
Original commit: elastic/x-pack-elasticsearch@9f26d557d0
This introduces a check to only pause the execution of watcher
when there is no metadata but there was a shard on this node before
that inside of the ClusterStateListener.
This prevents repeated logging that watcher was paused even
though it was not necessary to call anything.
Original commit: elastic/x-pack-elasticsearch@8d3a829ffb
Watcher had an undocumented feature to configure the settings of any
index template via updating the cluster settings. Instead of changing
the template one could add a setting during runtime that overwrote
what is written in the index template.
As index template are created once and not overwritten, users should
just change the index template - or the concrete index settings like
number of replicas.
This feature was not exposed in our documentation at all.
Original commit: elastic/x-pack-elasticsearch@32e1769925
This removes the Cluster State collector and resolver and moves the collection of the cluster state (and cluster health, which is already included in cluster stats).
This makes the tests a little more stable and removes an extra network hop during monitoring data collection.
Original commit: elastic/x-pack-elasticsearch@44851d2dd6
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
Original commit: elastic/x-pack-elasticsearch@b07aa78a7b
The aim of this change is to prevent many identical requests to create
watcher index templates being submitted when a cluster first starts up
and many cluster state updates are happening. Prior to this change, if
watcher's original index template creation requests queued up behind other
cluster state change requests then for each other request watcher would
re-request creation of all its index templates. After this change it
uses a strategy similar to that used by ML to only have one creation
request per index template in the cluster state change queue at any time.
Relates elastic/x-pack-elasticsearch#1368
Relates elastic/x-pack-elasticsearch#1631
Relates elastic/x-pack-elasticsearch#1650
Original commit: elastic/x-pack-elasticsearch@ad87bf3f78
Although the job stats for jobs with missing results indices are clearly
ruined, it's better to provide zeroes for the missing values and show the
stats for other jobs than to fail the whole request. This means the UI
can continue to function.
relates elastic/x-pack-elasticsearch#1656
Original commit: elastic/x-pack-elasticsearch@a06fa994a5
* Wait for job deletion if it is in the deleting state
* Tolerate errors if multiple force delete requests
Original commit: elastic/x-pack-elasticsearch@1f0c9fbb86
When a user executes a watch and specifies it as part of the
execute watch API, no triggered watch is created, as the watch
cannot be picked up anymore (it only leaves for the duration of
the request).
However until now the TriggeredWatchStore was invoked and tried
to delete this non-existing triggered watch, resulting in some
log cluttering.
This commit removes this try to delete a non-existing triggered
watch.
Original commit: elastic/x-pack-elasticsearch@3db125cea2
This is just the culmination of all of the minor PRs associated with 1068. It will:
- Drop the `.monitoring-data-N` index
- Drop use of `_type` in all cases (replaced by `doc` and a new `type` field)
- Drop the API version from the template name (e.g., instead of `.monitoring-es-6` we now use `.monitoring-es`).
- Change API version to `-6-` from `-2-`.
- Both exporters handle versioned resources (templates, pipelines, and watches)
- HTTP exporters will optionally (true by default) publish placeholders for the old, `-2` templates.
When this is backported, it will need to:
- Change `index_patterns` to `template` within the templates.
- Downgrade the version requirements for the templates, pipeline, and watches _and_ the HTTP exporter itself (all require 6.0)
This is a companion to the feature branch in X-Pack Kibana elastic/x-pack-kibana/pull/1318 and they need to be merged at the same time.
Original commit: elastic/x-pack-elasticsearch@6031cfffa4
This commit adds better security for scroll requests in that they are now tied to a single user as
we only authorize the request that creates the scroll. This is accomplished by adding a
SearchOperationListener that listens for new scroll contexts and stores the authentication on the
ScrollContext. Then upon
retrieval of the search context for a query or fetch, the current authentication is compared to the
authentication that was present when the scroll context was created. If the current authentication
belongs to a different user, then a SearchContextMissingException will be thrown to prevent leaking
a valid vs invalid scroll id.
Additionally, signing of a scroll id is only performed when there is a older node in the cluster
that would expect the scroll id to be signed. Once this is backported to 5.x, we can remove this
bwc layer for 6.0/master.
Original commit: elastic/x-pack-elasticsearch@0e5dcafd32
When a user or client intend to delete a datafeed
and its job, there is benefit into ensuring the
datafeed has gracefully stopped (ie no data loss).
In constrast, the desired behaviour is to stop and
delete the datafeed as quickly as possible.
This change adds a force option to the delete
datafeed action. When the delete is forced,
the datafeed is isolated, its task removed and,
finally, the datafeed itself is removed from the
metadata.
relates elastic/x-pack-elasticsearch#1533
Original commit: elastic/x-pack-elasticsearch@5ae0168bf2
We try to install empty ML metadata as soon as possible after startup
if none exists. However, this still leaves a short gap when the cluster
is active with no ML metadata. To avoid problems, functions that use
the ML metadata should treat this situation as equivalent to having
empty ML metadata.
relates elastic/x-pack-elasticsearch#1643
Original commit: elastic/x-pack-elasticsearch@8f0e00cda8
ignoreAliases allows to resolve index expressions against concrete indices only, rather than against indices and aliases. It is used for now only in IndicesAliasesRequest and the indices resolution code in the security plugin needs to be adapted accordingly.
Original commit: elastic/x-pack-elasticsearch@ae964eade9
This commit switches over to two index aliases per job: one for reading
and one for writing. In the future this will allow the addition of a
rollover endpoint for ML results indices. (Rollover is still not possible
following this change, but the change to make it possible in the future
should not be a breaking change now.)
Relates elastic/x-pack-elasticsearch#1599
relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@d648f4631f
* Add force delete job option
* Can’t kill a process on a 5.4 node
* Address review comments
* Rename KillAutodetectAction -> KillProcessAction
* Review comments
* Cancelling task is superfluous after it has been killed
* Update docs
* Revert "Cancelling task is superfluous after it has been killed"
This reverts commit 576950e2e1ee095b38174d8b71de353c082ae953.
* Remove unnecessary TODOs and logic that doesn't alwasys force close
Original commit: elastic/x-pack-elasticsearch@f8c8b38217
This test would sometime leak threads.
The "Timer thread for LDAPConnection" is created by the unboundid SDK - closing the connection should force the thread to terminate
Original commit: elastic/x-pack-elasticsearch@bd58a17a59
The cluster state listener to decide if watcher should be reloaded was
assuming that no aliases could be used and thus wrongly could trigger
a reload, which could have lead to wrong test results.
During debugging I also added a reason for reloading and fixed another
wrong test assumption.
Also the listener does not rely on previous cluster state, but stores this
in instance variable, as we need to compare with local state and not the
previous cluster state.
Original commit: elastic/x-pack-elasticsearch@582783a66d
Includes:
- Extensive changes to "mapping roles" section
- New section for role mapping API
- Updates to LDAP/AD/PKI realms to refer to API based role mapping
- Updates to LDAP/AD realms: `unmapped_groups_as_roles` only looks at file-based mappings
- Updates to LDAP/AD realms: new setting for "metadata"
Original commit: elastic/x-pack-elasticsearch@6349f665f5
This avoids log spam about being unable to create new mappings in indices
that are set to only allow one type. (It doesn't actually have any effect
on the deletion, which was working before despite the failure to create new
mappings for the legacy types referenced by the delete request.)
relates elastic/x-pack-elasticsearch#1634
Original commit: elastic/x-pack-elasticsearch@061ce7acf1
Because:
1. It's pointless, as new detector_index values are assigned when an
analysis_config is parsed
2. It creates a backwards compatibility issue when upgrading from v5.4
Original commit: elastic/x-pack-elasticsearch@2f61aa457e
This is the x-pack side of the removal of `accumulateExceptions()` for both `TransportNodesAction` and `TransportTasksAction`.
There are occasional, random failures that occur during API calls that are silently ignored from the caller's perspective, which also leads to weird API responses that have no response and also no errors, which is obviously untrue.
Original commit: elastic/x-pack-elasticsearch@9b57321549
Detectors now have a field called detector_index. This is also now the
field that needs to be supplied when updating a detector. (Previously
it was simply index, which was confusing.)
When detectors are added to an analysis_config it will reassign
ascending detector_index values starting from 0. The intention is
never to allow deletion of detectors from an analysis_config, but
possibly to allow disabling them in the future. This ensures that
detector_index values in results will always tie up with detector_ids
in the detectors that created them.
relates elastic/x-pack-elasticsearch#1275
Original commit: elastic/x-pack-elasticsearch@20a660b07b
Dimitris Athanasiou
Alexander Reelsen
David Kyle
Jason Tedor
jaymode
Ryan Ernst
Chris Earle
Chris Earle
David Roberts
Tim Vernum
Jack Conradson
Igor Motov
Jay Modi
Martijn van Groningen
Luca Cavanna
Nik Everett