This PR uses a new extension point that's being added to Elasticsearch (see https://github.com/elastic/elasticsearch/pull/27603) so that the security plugin can filter the mappings fields returned by get index, get mappings, get field mappings and field capabilities API.
This effort aims at filtering information returned by API in the `indices/admin` category and field capabilities. It doesn't filter what the cluster state api returns as that is a cluster level operation.
One question is about backwards compatibility given that we would like to have this in 6.2. Shall we treat this as a bug as mappings should have been filtered before? Not sure if it's going to break existing integrations.
relates elastic/x-pack-elasticsearch#340
Original commit: elastic/x-pack-elasticsearch@d7e3fd3fa1
Before this was done it was easy to get into the situation where a
job created in 5.x with a default model memory limit of 4GB could not
be opened on any node in the cluster. Following this change this
problem will no longer occur for jobs that ran for a decent amount of
time on the old cluster.
relates elastic/x-pack-elasticsearch#3181
Original commit: elastic/x-pack-elasticsearch@cb029debba
This suite seems to be timing out on the CI slaves. Hopefully,
some extra time will stop the failures.
Original commit: elastic/x-pack-elasticsearch@95ba86be79
The watcher threadpool size was always five times the CPU core
count, resulting in a huge threadpool when with even 24 cores.
This changes the behaviour to be five times the number of cores
by default - as watcher is usually waiting on I/O you should have more
threads than cores, but it maxes out with 50 threads, unless the number
of available cores is higher than that.
relates elastic/x-pack-elasticsearch#3052
Original commit: elastic/x-pack-elasticsearch@eab5deb113
In order to support the source directory repo split, this commit
disables security for the regular integration tests.
The MonitoringSettingsFilterTests already existed as REST test, so
this test has been removed.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@519154dd5f
The pagerduty action allows to send contexts, which contains an array
of texts or images, each with a link.
The field of this data was named 'context' instead of 'contexts' and
thus those contects were never correctly parsed on the pagerduty side.
Unfortunately pagerduty accepts any JSON, thus this was not caught so
far.
This commit allows parsing of the old field name to retain BWC, but when
written out via toXContent, it will always use the 'contexts' field name.
relates elastic/x-pack-elasticsearch#3184
Original commit: elastic/x-pack-elasticsearch@50f0b65d56
* [DOCS] Refreshed ML screenshots
* [DOCS] Added screenshots for ML Data Visualizer
* [DOCS] Addressed feedback about data visualizer
* [DOCS] Fixed typo in ML tutorial
Original commit: elastic/x-pack-elasticsearch@2603536a93
This commit fixes the minimum value being smaller than the maximum value in a call to the
scaledRandomIntBetween, which causes the test to fail.
Original commit: elastic/x-pack-elasticsearch@da7d0ce0ce
When you click "delete" in the UI it force-deletes the datafeed then
force-deletes the job. For a datafeed doing lookback, this results
in a close followed very quickly by a kill on the autodetect process.
Depending on thread scheduling this could cause a lot of spurious
errors and exception traces to be logged.
This change prevents the log spam in this scenario.
relates elastic/x-pack-elasticsearch#3149
Original commit: elastic/x-pack-elasticsearch@091240f32a
Our rolling upgrade tests were failing on many machines due to using
the 5.x default of 4GB model_memory_limit, which then propagated forward
to 6.1+ even though the default is now lower.
Original commit: elastic/x-pack-elasticsearch@3b23d8fe9d
The IndexLifecycleManagerIntegTests has a method that tests concurrency and executes a large number
of requests in parallel. On some machines, this can actually overwhelm a thread pool and cause the
test to fail for the wrong reasons. This commit bounds the total number of requests to 100.
Original commit: elastic/x-pack-elasticsearch@07613ada51
Created a smoke-test-monitoring-with-watcher project that runs REST
tests with watcher enabled to ensure that the proper watcher are
installed either when the local or the HTTP exporter are set up.
Also removed two more watcher imports in the tests.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@0a9abc3185
This was due to a hardcoded job ID that wouldn't exist when two random
numbers generated by the test framework were both small.
Original commit: elastic/x-pack-elasticsearch@9529e6d280
* [Monitoring] Update Beats Template with Metrics
- remove source_node.timestamp
- includes mapping for beat type = Metricbeat
* remove metricbeat and xpack objects which are not used in the UI
* use long instead of integer type
Original commit: elastic/x-pack-elasticsearch@083b9cc575
* [DOCS] Refresh screenshots in ML tutorial
* [DOCS] Refreshed screenshots for single metric job
* [DOCS] Removed outdated index screenshot
Original commit: elastic/x-pack-elasticsearch@14f39c3091
This change adds some debug and trace logging when we look up role names, to explain how each role was resolved.
At the moment we have very little insight into how roles are being resolved which can make it difficult to diagnose some issues.
Original commit: elastic/x-pack-elasticsearch@1b3c246186
Register LoggingAuditTrail as a ClusterStateListener.
Avoid querying for the localNode while on the ClusterStateApplier thread,
which tripps assertion. This can happen when logging audit events that
originate from the system.
relates elastic/x-pack-elasticsearch#3057
Original commit: elastic/x-pack-elasticsearch@66bc59682d
If the KeyStoreWrapper is closed, then we cannot validate secure settings (because we no longer have access to them)
The Realm group setting uses the "validate" method to ensure that child settings are correct, but it must ignore secure settings as it might get called
after startup (e.g. during a settings diff)
Original commit: elastic/x-pack-elasticsearch@b30db6bc62
* [DOCS] Subdivided getting started with ML pages
* [DOCS] Added new getting started page to build.gradle
Original commit: elastic/x-pack-elasticsearch@968187b048
This commit uses SPI to get the list of system wide reserved roles in
security. It does not yet split out the RoleDescriptor to a common
location so the implementing services still depend on security. Each
role, however, only depends on its own feature as well as security.
ref elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@efebc3e5c8
* Use XPackRestIT as base class for XDocsClientYamlTestSuiteIT
* Remove the XPackRestTestCase class
* Address review comments
* Fix checkstyle checks
Original commit: elastic/x-pack-elasticsearch@c2a5e60c12
This change fixes a problem that would cause job deletion to fail if ANY
index had a block on it, e.g. read-only.
The problem was that we were requesting the job aliases be deleted from
ALL indices in the system due to a misunderstanding with the format of the
get_aliases response. This didn't usually cause any noticable effects, as
only the ML indices would have the aliases. But in the case of a read-only
index it would cause an error, leading to unnecessary failure of the job
deletion.
Fixeselastic/machine-learning-cpp#428
Original commit: elastic/x-pack-elasticsearch@a573f85a00
Following the changes of elastic/x-pack-elasticsearch#2975 the hard limit on the number of ML jobs
per node is no longer the only limiting factor. Additionally there is
now a limit based on the estimated memory usage of the jobs, and this is
expected to provide a more sensible limit that accounts for differing
resource requirements per job.
As a result, it makes sense to raise the default for the hard limit on
the number of jobs, on the assumption that the memory limit will prevent
the node becoming overloaded if an attempt is made to run many large jobs.
Increasing the hard limit will allow more small jobs to be run than was
previously the case by default.
Of course, this change to the default will have no effect for customers
who have already overridden the default in their config files.
Original commit: elastic/x-pack-elasticsearch@9fed1d1237
The issue here is that if the first search request fails
(initScroll), then we do not have a scroll_id. However,
in order to retry the search, we reset the scroll. That
involves clearing the current search, but since we do
not have a scroll_id, the clear scroll request fails.
We end up reporting the failure for the scroll clearing,
rather than the actual problem.
This commit fixes that by avoiding clearing the scroll
when the scroll_id is null.
relates elastic/x-pack-elasticsearch#3146
Original commit: elastic/x-pack-elasticsearch@b5086028f6
This commit removes the KnownActionsTests as it no longer serves the intended purpose of catching
actions/handlers added to elasticsearch that security needs to be aware of. Today, it is common
for this test to break the build and as a mechanical response developers just add to the actions
or handlers file to get the build green again.
Relates elastic/x-pack-elasticsearch#1489
Original commit: elastic/x-pack-elasticsearch@0bdb5341f5
In order to support the repository split, this changes the
`AbstractWatcherIntegrationTestCase` to not run with security enabled.
We have a dedicated QA project called `smoke-test-watcher-with-security`,
where tests that explicitely need security should be running.
This commit removes the possibility to enable security as part of the
test case. In addition some tests have been moved over to the dedicated
project.
In addition the `timewarp` functionality cannot be configured with a
system property anymore. This would not have worked anyway, because
tests were already dependent on that functionality and did not have any
other means of running. A bit of redundant code was removed due to this.
Relates elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@b24b365ad1
Nhat Nguyen
Luca Cavanna
Chris Earle
David Kyle
David Roberts
Dimitrios Athanasiou
Alexander Reelsen
Lisa Cawley
jaymode
Tim Sullivan
Albert Zaharovits
Tim Vernum
Hendrik Muhs
Michael Basnight
Igor Motov
Jay Modi
Adrien Grand