This is just the culmination of all of the minor PRs associated with 1068. It will:
- Drop the `.monitoring-data-N` index
- Drop use of `_type` in all cases (replaced by `doc` and a new `type` field)
- Drop the API version from the template name (e.g., instead of `.monitoring-es-6` we now use `.monitoring-es`).
- Change API version to `-6-` from `-2-`.
- Both exporters handle versioned resources (templates, pipelines, and watches)
- HTTP exporters will optionally (true by default) publish placeholders for the old, `-2` templates.
When this is backported, it will need to:
- Change `index_patterns` to `template` within the templates.
- Downgrade the version requirements for the templates, pipeline, and watches _and_ the HTTP exporter itself (all require 6.0)
This is a companion to the feature branch in X-Pack Kibana elastic/x-pack-kibana/pull/1318 and they need to be merged at the same time.
Original commit: elastic/x-pack-elasticsearch@6031cfffa4
This commit adds better security for scroll requests in that they are now tied to a single user as
we only authorize the request that creates the scroll. This is accomplished by adding a
SearchOperationListener that listens for new scroll contexts and stores the authentication on the
ScrollContext. Then upon
retrieval of the search context for a query or fetch, the current authentication is compared to the
authentication that was present when the scroll context was created. If the current authentication
belongs to a different user, then a SearchContextMissingException will be thrown to prevent leaking
a valid vs invalid scroll id.
Additionally, signing of a scroll id is only performed when there is a older node in the cluster
that would expect the scroll id to be signed. Once this is backported to 5.x, we can remove this
bwc layer for 6.0/master.
Original commit: elastic/x-pack-elasticsearch@0e5dcafd32
When a user or client intend to delete a datafeed
and its job, there is benefit into ensuring the
datafeed has gracefully stopped (ie no data loss).
In constrast, the desired behaviour is to stop and
delete the datafeed as quickly as possible.
This change adds a force option to the delete
datafeed action. When the delete is forced,
the datafeed is isolated, its task removed and,
finally, the datafeed itself is removed from the
metadata.
relates elastic/x-pack-elasticsearch#1533
Original commit: elastic/x-pack-elasticsearch@5ae0168bf2
We try to install empty ML metadata as soon as possible after startup
if none exists. However, this still leaves a short gap when the cluster
is active with no ML metadata. To avoid problems, functions that use
the ML metadata should treat this situation as equivalent to having
empty ML metadata.
relates elastic/x-pack-elasticsearch#1643
Original commit: elastic/x-pack-elasticsearch@8f0e00cda8
ignoreAliases allows to resolve index expressions against concrete indices only, rather than against indices and aliases. It is used for now only in IndicesAliasesRequest and the indices resolution code in the security plugin needs to be adapted accordingly.
Original commit: elastic/x-pack-elasticsearch@ae964eade9
This commit switches over to two index aliases per job: one for reading
and one for writing. In the future this will allow the addition of a
rollover endpoint for ML results indices. (Rollover is still not possible
following this change, but the change to make it possible in the future
should not be a breaking change now.)
Relates elastic/x-pack-elasticsearch#1599
relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@d648f4631f
* Add force delete job option
* Can’t kill a process on a 5.4 node
* Address review comments
* Rename KillAutodetectAction -> KillProcessAction
* Review comments
* Cancelling task is superfluous after it has been killed
* Update docs
* Revert "Cancelling task is superfluous after it has been killed"
This reverts commit 576950e2e1ee095b38174d8b71de353c082ae953.
* Remove unnecessary TODOs and logic that doesn't alwasys force close
Original commit: elastic/x-pack-elasticsearch@f8c8b38217
This test would sometime leak threads.
The "Timer thread for LDAPConnection" is created by the unboundid SDK - closing the connection should force the thread to terminate
Original commit: elastic/x-pack-elasticsearch@bd58a17a59
The cluster state listener to decide if watcher should be reloaded was
assuming that no aliases could be used and thus wrongly could trigger
a reload, which could have lead to wrong test results.
During debugging I also added a reason for reloading and fixed another
wrong test assumption.
Also the listener does not rely on previous cluster state, but stores this
in instance variable, as we need to compare with local state and not the
previous cluster state.
Original commit: elastic/x-pack-elasticsearch@582783a66d
Includes:
- Extensive changes to "mapping roles" section
- New section for role mapping API
- Updates to LDAP/AD/PKI realms to refer to API based role mapping
- Updates to LDAP/AD realms: `unmapped_groups_as_roles` only looks at file-based mappings
- Updates to LDAP/AD realms: new setting for "metadata"
Original commit: elastic/x-pack-elasticsearch@6349f665f5
* [DOCS] Clarify ML node settings re transport requests
* [DOCS] Clarify xpack.ml.enabled based on feedback
Original commit: elastic/x-pack-elasticsearch@3102d1e3f3
This avoids log spam about being unable to create new mappings in indices
that are set to only allow one type. (It doesn't actually have any effect
on the deletion, which was working before despite the failure to create new
mappings for the legacy types referenced by the delete request.)
relates elastic/x-pack-elasticsearch#1634
Original commit: elastic/x-pack-elasticsearch@061ce7acf1
Because:
1. It's pointless, as new detector_index values are assigned when an
analysis_config is parsed
2. It creates a backwards compatibility issue when upgrading from v5.4
Original commit: elastic/x-pack-elasticsearch@2f61aa457e
This is the x-pack side of the removal of `accumulateExceptions()` for both `TransportNodesAction` and `TransportTasksAction`.
There are occasional, random failures that occur during API calls that are silently ignored from the caller's perspective, which also leads to weird API responses that have no response and also no errors, which is obviously untrue.
Original commit: elastic/x-pack-elasticsearch@9b57321549
Detectors now have a field called detector_index. This is also now the
field that needs to be supplied when updating a detector. (Previously
it was simply index, which was confusing.)
When detectors are added to an analysis_config it will reassign
ascending detector_index values starting from 0. The intention is
never to allow deletion of detectors from an analysis_config, but
possibly to allow disabling them in the future. This ensures that
detector_index values in results will always tie up with detector_ids
in the detectors that created them.
relates elastic/x-pack-elasticsearch#1275
Original commit: elastic/x-pack-elasticsearch@20a660b07b
At the end of the test, LocalExporterTests checks if no more monitoring
data are exporter by checking multiple times the last time nodes_stats
documents were exported, stopping after 10 seconds. It does this in a
@After annotated method but it would be better to do this in a finally
block. Also, it should search for node_stats documents only if the
monitoring indices exist and are searchable to avoid some "all shards
failed" failures.
Original commit: elastic/x-pack-elasticsearch@90ffb4affd
* [DOCS] Add details about ML count functions
* [DOCS] Address feedback in ML count functions
* [DOCS] Clarify ML population analysis in non-zero count functions
Original commit: elastic/x-pack-elasticsearch@24dbeba891
The kibana user role in mixed cluster test expects a old cluster test to have run first that would
disable the kibana user, but that test is not executed on 5.1.1 and prior versions. This change
makes a update request an upsert to account for this.
relates elastic/x-pack-elasticsearch#1527
Original commit: elastic/x-pack-elasticsearch@df2945610e
Now we've set the option for one type per index it causes a stack trace
in to be logged if we issue a request to delete two documents with
different types. We only do this to cover the case of documents left
over from v5.4. We can avoid it by deleting by query using just the
document IDs.
Original commit: elastic/x-pack-elasticsearch@2abffc7d95
This commit removes ClientProxy and WatcherClientProxy classes. They
were added in times, where there were issues with guice and circular
dependencies. However there is no guice anymore and on top of that
the classes do not add any value.
We can switch to use a regular client, but have to make sure that
the InternalClient is injected in all the transport actions as those
is able to query data, when security is enabled.
Original commit: elastic/x-pack-elasticsearch@763a79b2f7
The goal of this change is to allow datafeeds to start
when the job is in the opening state. This makes the API
more async and it allows clients like the ML UI to open a
job and start its datafeed without having to manage the
complexity of dealing with timeouts due to the job taking
time to open due to restoring a large state.
In order to achieve this, this commit does a number of things:
- accepts a start datafeed request when the job is opening
- adds logic to the DatafeedManager to wait before running the
datafeed task until the job is opened
- refactord the datafeed node selection logic into its own class
- splitd selection issues in critical and non-critical with regard
to creating the datafeed task
- refactord the unit tests to make simpler to write & understand
- adds unit tests for added and modified functionality
- changes the response when the datafeed cannot be started to
be a conflict exception
relates elastic/x-pack-elasticsearch#1535
Original commit: elastic/x-pack-elasticsearch@c83196155d
In preparation for the removal of types, new security types like invalidated-tokens are stored in the .security
index under the generic "doc" type, with a query filter on `doc_type`.
In order to avoid id clashes, we also need to use that doc_type as part of the document id.
relates elastic/x-pack-elasticsearch#1300
Original commit: elastic/x-pack-elasticsearch@469724a228
When an error response contains multiple layers of errors, Kibana displays
the one labelled root_cause. The definition of root_cause is the most
deeply nested ElasticsearchException. Therefore, it is of great benefit to
the UI if our config validation returns the actual problem in an
ElasticsearchException rather than an IllegalArgumentException.
This commit also adds an extra validation check to catch the case of a
single job config containing fields x.y as well as x earlier. Previously
this was caught when we tried to create results mappings, and was
accompanied by an error suggesting that using a dedicated results index
would help, when clearly it won't for a clash in a single job config.
Fixeselastic/x-pack-kibana#1387Fixeselastic/prelert-legacy#349
Original commit: elastic/x-pack-elasticsearch@7d1b7def6c
Otherwise it's possible that the get_filter endpoint can return a filter that's been
deleted. Although this is the behaviour of the search API, specific metadata
management APIs should provide better guarantees.
Original commit: elastic/x-pack-elasticsearch@818495f176
This allows us to build both 5.5.0-SNAPSHOT and 5.4.1-SNAPSHOT
artifacts for backwards compatibility testing. It is a port of
elastic/elasticsearch:24870 to x-pack and will be super useful
when elastic/elasticsearch:24846 is ported to x-pack.
Original commit: elastic/x-pack-elasticsearch@0ea443f488
Previously there were two @After methods in the XPackRestIT class, and
there is no guarantee about the order in which these run. This commit
replaces these with a single @After method that calls the cleanup methods
in a well-defined order.
Original commit: elastic/x-pack-elasticsearch@d3ab366591
The has_privileges API now supports wildcards.
The semantics are that the user must have a superset of the wildcard being checked.
---------------------
Role | Check | Result
---------------------
* | foo* | true
f* | foo* | true
foo* | foo* | true
foo* | foo? | true
foo? | foo? | true
foo? | foo* | false
foo | foo* | false
Original commit: elastic/x-pack-elasticsearch@817550db17
We don't hyphenate metadata anywhere else.
Also added tests for the LdapMetaDataResolver as they were completely absent.
Original commit: elastic/x-pack-elasticsearch@eec647ba93
Igor Motov
jaymode
Chris Earle
Jay Modi
Martijn van Groningen
Dimitris Athanasiou
David Roberts
Luca Cavanna
David Kyle
Tim Vernum
Alexander Reelsen
Lisa Cawley
Clinton Gormley
Ryan Ernst
Nik Everett
Colin Goodheart-Smithe
Tanguy Leroux
markharwood
David Pilato
Suyog Rao