This removes the creation and handling of the Beats monitoring template and its data until we actually expect to support it (most likely 6.2 - 6.3).
Original commit: elastic/x-pack-elasticsearch@2dc8abbb37
* [Monitoring/Cluster Alerts] Fix the email message for cluster license expiration resolved
* fix making payload.message show only when new
Original commit: elastic/x-pack-elasticsearch@6d54b02913
When the datafeed uses aggregations and in order to accommodate
derivatives, an extra bucket is queried at the beginning of each
search. In order to avoid visiting the same bucket twice, we need
to search buckets aligned to the histogram interval. This allows
us to steer away from partial buckets, and thus avoid the problem
of dropping or duplicating data.
relates elastic/x-pack-elasticsearch#2519
Original commit: elastic/x-pack-elasticsearch@e03dde5fea
There are two rest tests that rely on hardcoded license
signatures that use the dev public key. These tests fail
when tests are run with build.snapshot=false. This Commit
blacklists these two tests in that scenario
relates elastic/x-pack-elasticsearch#2527.
Original commit: elastic/x-pack-elasticsearch@7581e8d699
Some model plot documents should not have an "actual" value, for example
when no input events were seen for a meean/min/max detector in a particular
bucket. Prior to this change we would set the "actual" value to 0 for such
model plot documents. Following this change no "actual" value will be
present in these documents.
Only newly created model plot documents are affected. Model plot documents
that were incorrectly written in the past will remain wrong forever.
relates elastic/x-pack-elasticsearch#2528
Original commit: elastic/x-pack-elasticsearch@47a7365f59
The problem here is that the code was ignoring buckets
whose start time was before the start time of the extractor.
However, this is not a good enough condition. For example,
when there are no data in the bucket extra bucket that is
being queried, the first bucket will be the one containing
the start time.
This commit fixes the issue by changing the condition to
ignore buckets before the first bucket that includes the
start time of the extraction.
relates elastic/x-pack-elasticsearch#2519
Original commit: elastic/x-pack-elasticsearch@15c7d2655f
If the keystore type is not explicitly specified, infer it from the filename.
Treats .p12, .pfx and .pkcs12 as being PKCS12, all others as jks.
This will allow certgen to produce PKCSelastic/x-pack-elasticsearch#12 files by default and make it easy to use them as x-pack keystores
Original commit: elastic/x-pack-elasticsearch@fc361f0d87
Beginning with 7.0, the cleaner service will no longer automatically cleanup .marvel indices regardless of their age.
Original commit: elastic/x-pack-elasticsearch@5b90e6f62a
This changes Monitoring's Cleaner Service to remove any legacy Monitoring index that is appropriately old.
This includes any `.marvel-*` index and also the "data" indices used by both Marvel and 5.0 - 5.4 versions of X-Pack monitoring, as well as the legacy alerts index.
Original commit: elastic/x-pack-elasticsearch@8d99f5518b
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.
Original commit: elastic/x-pack-elasticsearch@d66145de54
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.
Relates elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@abc66ec67d
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
This change removes security index access from the xpack user by creating its own specific role
and adds a xpack security user that maintains the superuser role so that it can perform all
operations necessary for security.
Original commit: elastic/x-pack-elasticsearch@ad906bc913
This change will enforce transport SSL to be enforced if security is enabled and the
license in the clusterstate is a production license. The cluster state is loaded from
local storage such that we don't need to join a cluster to make these checks. Yet, the cluster
might have already got a different license if the node got disconnected while the license got
downgraded and then TLS got disabled. This corner case requires manual intervention which
we consider ok given the simplicity of this change.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@5765b7cd21
integrate forecasting feature branch into master
- add endpoint xpack/ml/job/forecast to request forecasting on data of ml-jobs
- current parameters: end time
- persists forecast results into shared or own index
- different runs are separated by a 'forecast id'
relates elastic/x-pack-elasticsearch#1838
Original commit: elastic/x-pack-elasticsearch@f9d701a6bc
After the addition of the secure settings in 5.6, the truststore.password setting for the PKI realm
was no longer registered. This would cause new nodes to fail for customers that were upgrading and
had configured a PKI realm with a truststore. This change registers the setting and adds a test to
ensure a realm configuration with the old setting passes validation.
Relates elastic/support-dev-help#2505
Original commit: elastic/x-pack-elasticsearch@54da044a27
This change prevents a node from joining a cluster with a production license (gold, platinum, standard) iff the cluster doesn't have TLS setup. This is mainly a BWC oriented change that prevents joining old 5.x clusters without a TLS setup.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@21f5a58472
Currently the maintenance task is executed at 30 minutes past
midnight of each day. In the scenario where multiple clusters
are running on the same hardware infrastructure they all will
be running at the same time, competing for resources.
This commit changes this by adding a random offset to the
execution time which ranges from 0 to 119 minutes. The
minute granularity means that different offsets give at
least 1 minute for the maintenance task to end. Moreover,
the 2 hour window gives enough slots for different offsets
to occur and remains within what most people would think
as "middle of the night".
relates elastic/x-pack-elasticsearch#2273
Original commit: elastic/x-pack-elasticsearch@b538923aca
Changes the default query delay from 1m to a random
value between 1m and 2m. The motivation is to avoid
having multiple jobs firing their searches at the same
time which may potentially lead to increased load
on the machine.
relates elastic/x-pack-elasticsearch#2472
Original commit: elastic/x-pack-elasticsearch@3224e836fa
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.
Original commit: elastic/x-pack-elasticsearch@37ba90359e
As there are no master node operations anymore.
* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed
Original commit: elastic/x-pack-elasticsearch@4607dd538c
The datafeed runs on frequency-aligned intervals behind
query_delay. Currently, when a real-time run is triggered,
we subtract query_delay from now and then we take the aligned
interval. This results into running frequency + query_delay
behind now. The fix involves simply adding the query_delay
into the time real-time runs occur.
Relates elastic/x-pack-elasticsearch#2426
Original commit: elastic/x-pack-elasticsearch@61ceaaca8f
"Established" memory use will be one of the building blocks for smarter node
allocation.
In order for a job to be considered to have established memory usage it must:
- Have generated at least 20 buckets of results
- Have generated at least one model size stats document
- Have low variability of model bytes in model size stats documents in the
time period covered by the last 20 buckets, which is defined as having a
coefficient of variation of no more than 0.1
Relates elastic/x-pack-elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@5032eb01d8
The changes made for elastic/x-pack-elasticsearch#2369 showed that the ML security tests were seriously
weakened by the decision to grant many "minimal" privileges to all users
involved in the tests. A better solution is to override the auth header
such that a superuser runs setup actions and assertions that work by
querying raw documents in ways that an end user wouldn't. Then the ML
endpoints can be called with the privileges provided by the ML roles and
nothing else.
Original commit: elastic/x-pack-elasticsearch@4de42d9e54
When watcher is loading it must only load the watches
which are active instead of all possible watches.
This loading happens on start up as well as when shards
relocate.
Original commit: elastic/x-pack-elasticsearch@29df56b99d
Implementation details of ML endpoints should be performed using the
internal client, so that the end user only requires permissions for
the public ML endpoints and does not need to know how they are
implemented. This change fixes some instances where this rule was
not adhered to.
Original commit: elastic/x-pack-elasticsearch@01c8f5172c
* Add support for authz checks at on shard requests
* Add Rest Tests for authorization
* Bulk security - Only reject individual items, rather than a whole shard
* Sync with core change
* Grant "delete" priv in ML smoketest
This role had index and+bulk privileges but it also needs delete (in order to delete ML model-snapshots)
Original commit: elastic/x-pack-elasticsearch@830e89e652
The method to check if watcher was enabled was returning
`randomBoolean()` and thus could change during test runs.
This fixes the test to ensure that always the same value
is returned and documents this requirement.
relates elastic/x-pack-elasticsearch#1783
Original commit: elastic/x-pack-elasticsearch@97bf3cfc29
This parameter ceased to work when Elasticsearch 5 introduced strict
parameter handling, because of a missing test.
This commit adds the parameter to the rest handler responseParams()
and adds a test along with the needed YAML definition.
relates elastic/x-pack-elasticsearch#2396
Original commit: elastic/x-pack-elasticsearch@8638df336c
If the duration time was 0 (and this might happen due to
using System.currentTimeMillis), the is_true check
still returns false.
The correct fix will be done later to replace the offending
measurement calls and replace them. Then we can add back this
line.
Original commit: elastic/x-pack-elasticsearch@076a9a37cc
A bunch of integration tests should have been built as unit tests
or already have unit test equivalents.
This commit removes integration tests as well as adding REST equivalents
or creating unit tests instead of extending from AbstractWatcherIntegrationTestCase
Original commit: elastic/x-pack-elasticsearch@a97b99467d
* Don’t count incomplete buckets in data stream diagnostics
* Fix tests now bucket_count doesn’t include partial buckets
Original commit: elastic/x-pack-elasticsearch@bc1a7bd9e7
This commit adapts to the renaming of the TransportResyncReplicationAction in core and also adds
an assertion to the check for a user being present when sending a request. The assertion is added
so that we can hopefully catch these scenarios in our testing as the assertion error will cause the
node to die but the ISE will just be seen in the logs. Since we do not run with assertions enabled
in production, the ISE is left to handle those cases.
relates elastic/x-pack-elasticsearch#2335
Original commit: elastic/x-pack-elasticsearch@c5ce0c93af
In `mutateInstance()` the from or size could become negative if the other one was pushed over the limit for `from + size`. This change fixes this case to make sure after the mutate method is called the from and size obey the limit but are also both `>= 0`
relates elastic/x-pack-elasticsearch#2344
Original commit: elastic/x-pack-elasticsearch@a8a7072fcc
* Add reserved dashboards_only_user role
* Fix line too long
* add tests for new reserved role
* rename role, hopefully fix tests
* Fix test
Original commit: elastic/x-pack-elasticsearch@99f6718c7c
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.
* TokenService default enabled if HTTP_ENABLED == false
* Fixed tests that need TokenService explicitly enabled
* [DOC] Default value for `xpack.security.authc.token.enabled`
Original commit: elastic/x-pack-elasticsearch@bd154d16eb
* Moves more classes over to ToXContentObject/Fragment
* Removes ToXContentToBytes
* Removes ToXContent from Enums
* review comment fix
* slight change to use XContantHelper
Original commit: elastic/x-pack-elasticsearch@0f2d3f328b
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.
Relates to elastic/elasticsearch#25859
Original commit: elastic/x-pack-elasticsearch@4bf266e0b0
As there are two indices to upgrade for watcher, it makes a lot of sense
to also have two upgrade checks.
There is one upgrader for the watches index, which deletes
old templates, adds the new one before and then does the reindexing.
Same for the triggered watches index.
This also means, that there will be two entries popping up in the kibana
UI.
Note: Each upgrade check checks if the other index (for the .watches
upgrade check the triggered watches index and vice versa) is already
upgraded and only if that is true, watcher is restarted.
relates elastic/x-pack-elasticsearch#2238
Original commit: elastic/x-pack-elasticsearch@2c92040ed6
I noticed this while working on a previous issue with atomic move writer
(silent swallowing of exceptions). Namely, atomic move writer has
dangerous semantics. The problem is as follows: atomic move writer works
by writing lines to a temporary file, and then in its close method it
replaces the target path with the temporary file. However, the close
method is invoked whether or not all writes to the temporary file
succeeded (because writers obtained from atomic move writer are used in
try-with-resources blocks, as they should be). There is no way to
distinguish that the writer is being closed in a successful scenario
versus a failure scenario. In the close method for atomic move writer,
the target file is replaced by the temporary file. This means that the
target file is replaced whether or not writing to the temporary file
actually succeeded. Since these atomic move writers are used for user
configuration files (users and user_roles), a failure here can lead to
data loss for the user, a tragedy!
There is another (less serious) problem with the atomic move
writer. Since the close method tries to move the temporary file in place
of the existing file, the temporary file can be left behind if there is
another failure in the close method (e.g., closing the underlying file
after writing, or setting the permissions on the temporary file). This
means that in some situations, atomic move writer will leave temporary
files behind (which is not definitively not atomic).
This commit replaces the atomic move writer with a safer mechanism. We
still perform the write atomically in the following sense: we write to a
temporary file. Either writing to that file succeeds or it fails. If
writing succeeds, we replace the existing file with the temporary
file. If writing fails, we clean up the temporary file and the existing
file remains in place.
Relates elastic/x-pack-elasticsearch#2299
Original commit: elastic/x-pack-elasticsearch@3199decb0a
Certain types of datafeeds cannot have null chunking configs, so
setting chunking config to null sometimes doesn't stick as null
Original commit: elastic/x-pack-elasticsearch@3a52bad460
When a watch is executed currently, it gets passed an in-memory
watch object, that was loaded, before the execution started.
This means there is a window of time, where an old watch could still
be executing, then a watch gets loaded for execution, then the old watch
execution finishes and updates the watch status and thus reindexes the
watch.
Now the watch, that got loaded for execution, executes and tries to
store its watch status, but fails, because the version of the watch
has changed.
This commit changes the point in time where the watch is loaded. Now
this only happens, while a watch is in its protected execution block,
and thus we can be sure, that there is no other execution of the watch
happening.
This will primarily impact watches, that execute often, but their
runtime is longer than the configured interval between executions.
Side fix: Removed some duplicate testing method and moved into
WatcherTestUtils, fixed a tests with a ton of if's with random booleans
into separate tests.
relates elastic/x-pack-elasticsearch#395
Original commit: elastic/x-pack-elasticsearch@bf393023d7
Today it's impossible to run xpack bwc tests against any other branch
or remote than upstream. This allows to pass `-Dtests.bwc.refspec` to
change the refspec to use for the bwc tests.
Original commit: elastic/x-pack-elasticsearch@4d365f5a6e
Tanguy Leroux
Chris Earle
David Kyle
Albert Zaharovits
Tim Sullivan
Igor Motov
Dimitrios Athanasiou
Hendrik Muhs
Yannick Welsch
Tal Levy
Nik Everett
David Roberts
Tim Vernum
Suyog Rao
Simon Willnauer
Tim Brooks
jaymode
Michael Basnight
Andy Bristol
Jay Modi
Jason Tedor
Alexander Reelsen
Ryan Ernst
Jim Ferenczi
Colin Goodheart-Smithe
Christoph Büscher
Stacey Gammon