This changes Monitoring's Cleaner Service to remove any legacy Monitoring index that is appropriately old.
This includes any `.marvel-*` index and also the "data" indices used by both Marvel and 5.0 - 5.4 versions of X-Pack monitoring, as well as the legacy alerts index.
Original commit: elastic/x-pack-elasticsearch@8d99f5518b
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.
Original commit: elastic/x-pack-elasticsearch@d66145de54
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.
Relates elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@abc66ec67d
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
* Move CLI to TransportSqlAction
* Moves REST endpoint from `/_cli` to `/_sql/cli`
* Removes the special purpose CLI transport action instead
implements the CLI entirely on the REST layer, delegating
all SQL stuff to the same action that backs the `/_sql` REST
API.
* Reworks "embedded testing mode" to use a `FilterClient` to
bounce capture the sql transport action and execute in embedded.
* Switches CLI formatting from consuming the entire response
to consuming just the first page of the response and returning
a `cursor` that can be used to read the next page. That read is
not yet implemented.
* Switch CLI formatting from the consuming the `RowSetCursor` to
consuming the `SqlResponse` object.
* Adds tests for CLI formatting.
* Support next page in the cli
* Rename cli's CommandRequest/CommandResponse to
QueryInitRequest/QueryInitResponse to line up with jdbc
* Implement QueryPageRequest/QueryPageResponse in cli
* Use `byte[]` to represent the cursor in the cli. Those bytes
mean something, but only to the server. The only reasonint that
the client does about them is "if length == 0 then there isn't a
next page."
* Pull common code from jdbc's QueryInitRequest, QueryPageRequest,
QueryInitResponse, and QueryPageResponse into the shared-proto
project
* By implication this switches jdbc's QueryPageRequest to using
the same cursor implementation as the cli
Original commit: elastic/x-pack-elasticsearch@193586f1ee
This change removes security index access from the xpack user by creating its own specific role
and adds a xpack security user that maintains the superuser role so that it can perform all
operations necessary for security.
Original commit: elastic/x-pack-elasticsearch@ad906bc913
This change will enforce transport SSL to be enforced if security is enabled and the
license in the clusterstate is a production license. The cluster state is loaded from
local storage such that we don't need to join a cluster to make these checks. Yet, the cluster
might have already got a different license if the node got disconnected while the license got
downgraded and then TLS got disabled. This corner case requires manual intervention which
we consider ok given the simplicity of this change.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@5765b7cd21
integrate forecasting feature branch into master
- add endpoint xpack/ml/job/forecast to request forecasting on data of ml-jobs
- current parameters: end time
- persists forecast results into shared or own index
- different runs are separated by a 'forecast id'
relates elastic/x-pack-elasticsearch#1838
Original commit: elastic/x-pack-elasticsearch@f9d701a6bc
After the addition of the secure settings in 5.6, the truststore.password setting for the PKI realm
was no longer registered. This would cause new nodes to fail for customers that were upgrading and
had configured a PKI realm with a truststore. This change registers the setting and adds a test to
ensure a realm configuration with the old setting passes validation.
Relates elastic/support-dev-help#2505
Original commit: elastic/x-pack-elasticsearch@54da044a27
This change prevents a node from joining a cluster with a production license (gold, platinum, standard) iff the cluster doesn't have TLS setup. This is mainly a BWC oriented change that prevents joining old 5.x clusters without a TLS setup.
Relates to elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@21f5a58472
Currently the maintenance task is executed at 30 minutes past
midnight of each day. In the scenario where multiple clusters
are running on the same hardware infrastructure they all will
be running at the same time, competing for resources.
This commit changes this by adding a random offset to the
execution time which ranges from 0 to 119 minutes. The
minute granularity means that different offsets give at
least 1 minute for the maintenance task to end. Moreover,
the 2 hour window gives enough slots for different offsets
to occur and remains within what most people would think
as "middle of the night".
relates elastic/x-pack-elasticsearch#2273
Original commit: elastic/x-pack-elasticsearch@b538923aca
Changes the default query delay from 1m to a random
value between 1m and 2m. The motivation is to avoid
having multiple jobs firing their searches at the same
time which may potentially lead to increased load
on the machine.
relates elastic/x-pack-elasticsearch#2472
Original commit: elastic/x-pack-elasticsearch@3224e836fa
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.
Original commit: elastic/x-pack-elasticsearch@37ba90359e
As there are no master node operations anymore.
* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed
Original commit: elastic/x-pack-elasticsearch@4607dd538c
The datafeed runs on frequency-aligned intervals behind
query_delay. Currently, when a real-time run is triggered,
we subtract query_delay from now and then we take the aligned
interval. This results into running frequency + query_delay
behind now. The fix involves simply adding the query_delay
into the time real-time runs occur.
Relates elastic/x-pack-elasticsearch#2426
Original commit: elastic/x-pack-elasticsearch@61ceaaca8f
"Established" memory use will be one of the building blocks for smarter node
allocation.
In order for a job to be considered to have established memory usage it must:
- Have generated at least 20 buckets of results
- Have generated at least one model size stats document
- Have low variability of model bytes in model size stats documents in the
time period covered by the last 20 buckets, which is defined as having a
coefficient of variation of no more than 0.1
Relates elastic/x-pack-elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@5032eb01d8
Instead of throwing and catching an exception for invalid
indices this returns *why* they are invalid in a convenient
object form that can be thrown as an exception when the index
is required or the entire index can be ignored when listing
indices.
Original commit: elastic/x-pack-elasticsearch@f45cbce647
This integrates SQL's metadata calls with security by creating
`SqlIndicesAction` and routing all of SQL's metadata calls through
it. Since it *does* know up from which indices it is working against
it can be an `IndicesRequest.Replaceable` and integrate with the
existing security infrastructure for filtering indices.
This request is implemented fairly similarly to the `GetIndexAction`
with the option to read from the master or from a local copy of
cluster state. Currently SQL forces it to run on the local copy
because the request doesn't properly support serialization. I'd
like to implement that in a followup.
Original commit: elastic/x-pack-elasticsearch@15f9512820
The changes made for elastic/x-pack-elasticsearch#2369 showed that the ML security tests were seriously
weakened by the decision to grant many "minimal" privileges to all users
involved in the tests. A better solution is to override the auth header
such that a superuser runs setup actions and assertions that work by
querying raw documents in ways that an end user wouldn't. Then the ML
endpoints can be called with the privileges provided by the ML roles and
nothing else.
Original commit: elastic/x-pack-elasticsearch@4de42d9e54
When watcher is loading it must only load the watches
which are active instead of all possible watches.
This loading happens on start up as well as when shards
relocate.
Original commit: elastic/x-pack-elasticsearch@29df56b99d
Implementation details of ML endpoints should be performed using the
internal client, so that the end user only requires permissions for
the public ML endpoints and does not need to know how they are
implemented. This change fixes some instances where this rule was
not adhered to.
Original commit: elastic/x-pack-elasticsearch@01c8f5172c
* Add support for authz checks at on shard requests
* Add Rest Tests for authorization
* Bulk security - Only reject individual items, rather than a whole shard
* Sync with core change
* Grant "delete" priv in ML smoketest
This role had index and+bulk privileges but it also needs delete (in order to delete ML model-snapshots)
Original commit: elastic/x-pack-elasticsearch@830e89e652
The method to check if watcher was enabled was returning
`randomBoolean()` and thus could change during test runs.
This fixes the test to ensure that always the same value
is returned and documents this requirement.
relates elastic/x-pack-elasticsearch#1783
Original commit: elastic/x-pack-elasticsearch@97bf3cfc29
This parameter ceased to work when Elasticsearch 5 introduced strict
parameter handling, because of a missing test.
This commit adds the parameter to the rest handler responseParams()
and adds a test along with the needed YAML definition.
relates elastic/x-pack-elasticsearch#2396
Original commit: elastic/x-pack-elasticsearch@8638df336c
