With the new template and mapping update mechanisms introduced as part
of the Upgrade API work, the NativeRealmMigrator is no longer needed or
used. This commit removes the NativeRealmMigrator code and the
associated tests for it.
Original commit: elastic/x-pack-elasticsearch@5d2d7a582c
This is related to elastic/x-pack-elasticsearch#1778. This commit adds a setting
(xpack.license.self_generated.type) which allows the user to specify
what type of license will be self-generated on node startup. The allowed
types are basic or trial.
Original commit: elastic/x-pack-elasticsearch@0a16a59e10
To achieve backwards compatibility the easiest way is
to restore the old watcher stats, which are supposed to run
on the master node only.
The distributed watcher stats have been moved under the statsdist
package and the action name has been changed as well.
This way there is no need to have a serialization BWC layer,
we can just call different actions.
Note: With the current approach developers still need to change
their java applications if they try to receive watcher stats,
as by default we are now using the distributed stats in the
watcher client.
Original commit: elastic/x-pack-elasticsearch@49b3a45452
Closing jobs can still use some or all of the threads that communicate
with the C++ process - the number of threads used will decrease as the
close progresses, but at the beginning of the closure all are still in
use. Therefore, to prevent the risk of EsRejectedExecution exceptions
for the autodetect communications threadpool, closing jobs need to be
considered when checking that enough threads exist to start a new
process. An explicit check produces a much more understandable error
message than an EsRejectedExecution exception.
relates elastic/x-pack-elasticsearch#1364
Original commit: elastic/x-pack-elasticsearch@845bfe0188
This is important for two reasons:
1. If a job hangs in the closing state for any reason there is now a
way to get rid of it
2. The force delete endpoint (as used by the UI) killed open jobs before
deleting them, but could not kill closing jobs, which created the
possibility that if a job was deleted from the UI while in the closing
state then the last few results could be indexed after the deletion
completed
relates elastic/x-pack-elasticsearch#1796
Original commit: elastic/x-pack-elasticsearch@1471106e06
Add an option to the ssl certificate generation tool (certgen) that generates PKCSelastic/x-pack-elasticsearch#12 (.p12) files in addition to the certificate (.crt) and key (.key) files.
A PKCSelastic/x-pack-elasticsearch#12 store is a container format for storing multiple crypto objects in a single file, which means we can put the cert and key into the same file.
These format is particularly useful for .NET environments, where .NET Core requires a single into file for PKI authentication.
Also adds documentation for all the command-line options in certgen.
Original commit: elastic/x-pack-elasticsearch@d10f88f12d
* [Monitoring] Add Cluster Alert for X-Pack License Expiration
* work on scripts round 1
* updates per feedback
* spaces
* fix NPE error in transform
* condition to allow updating metadata in the alerts index in every interval
* custom subject message
* update name of indexing actions
* ensure ctx.metadata is updated even if alert is not resolved
* fix omission of absoluteTime
* skip info-level alerts for trial-type license
* move break above `fromNow` declaration
* fix test
Original commit: elastic/x-pack-elasticsearch@f13718f5b5
the upgrade API is lacking some functionality in a special case,
where triggered_watches exists, but .watches does not. This
deletes the triggered watches index manually until we integrated
this properly in the upgrade API to fix the tests
Original commit: elastic/x-pack-elasticsearch@e9d1b0d35d
This commit adds the upgrade API functionality and script for security.
It also enables previously muted tests that would fail due to the lack
of security upgrade features in testing cluster restarts and old
security index backward compatibility.
Original commit: elastic/x-pack-elasticsearch@4abe9f1263
This change makes the restricted trust configuration delegate the list of files to monitor to the
trust configuration that it wraps so that all files that should be monitored for changes are
monitored for changes.
Relates elastic/x-pack-elasticsearch#1919
Original commit: elastic/x-pack-elasticsearch@227db92ac0
We were catching IndexNotFoundException, which was hiding the fact that delete index and update aliases APIs don't accept aliases anymore. Now that the exception changed this problem popped up. We now rather call get index providing .security as index name, then delete the concrete indices returned in the response.
Original commit: elastic/x-pack-elasticsearch@18f64f9a41
This change fixes some cases where calls to the LDAP library can result in a new connection being
created that were not wrapped in privileged connect calls. This would result in permission denied
errors when trying to make the connection.
Original commit: elastic/x-pack-elasticsearch@182c790dd4
This changes the SecurityTribeIT tests to use the same settings object when creating the settings
for the tribe node. Previously two different objects were being created and we would read regular
settings from one and secure settings from another. This causes problems since randomization means
that there may be settings added on the first call that do not get added on the second call. One
example is the randomization of when to only use a keystore or to use both a keystore and a
truststore. On the first call, we would add settings for both a keystore and a truststore but on
the second call only the keystore settings were added. This lead to failures as we would not be
able to open a password protected truststore since the password was never added to the secure
settings.
relates elastic/x-pack-elasticsearch#2005
Original commit: elastic/x-pack-elasticsearch@bbdb3ec662
This change clears the caches in the native realm and the composite roles store when there is a
a change in the health of the security index that necessitates this. When the security index goes
to a red state, the caches are left in tact as this allows for management operations to be
performed for a limited amount of time. When the index transitions out of the red state or exists
when it didn't exist before, the caches will be cleared so that we remove any stale values.
relates elastic/x-pack-elasticsearch#1789
Original commit: elastic/x-pack-elasticsearch@914959ea6b
This adds a bootstrap-check that makes it an error to configure a role mapping file that doesn't exist or cannot be parsed.
We are still lenient on dynamic reload because
(a) killing a running node is quite drastic
(b) file writes aren't atomic, so we might be picking up a file that is half way through being written (etc).
If you rely on the default role mapping filename, then it doesn't need to exist (because you might be using the role mapping API instead) but if it does exist it has to parse successfully
Original commit: elastic/x-pack-elasticsearch@5424dea4c4
The default for group_search.attribute is to search by DN, but explicitly setting that to dn
wouldn't work because the DN is returned in a special value in the result, and not in the attributes list.
This change detects when user_attribute is set to dn and treats it the same way as the default value.
Original commit: elastic/x-pack-elasticsearch@1933410a0b
- Changes the reloading test to always trust the "trusted" cert so that the health-ping works
- Adds some more logging in case we get new failures
Original commit: elastic/x-pack-elasticsearch@993bf9c721
This is related to elastic/x-pack-elasticsearch#1217. This commit requires that the elastic password
be bootstrapped for the user to be authenticated. As a result it removes
the special "setup" mode that allowed the user to be authenticated from
localhost.
Additionally, this commit updates the tests to work with this
functionality.
Original commit: elastic/x-pack-elasticsearch@d0d5d697a7
When a user asks for upgrade information for all indices and there are no indices in the cluster, upgrade assistance should just return an empty response indicating that no indices require upgrade or reindexing. This commit also reverts the temporary fix in WatchBackwardsCompatibilityIT tests that was added as a workaround for this issue.
Original commit: elastic/x-pack-elasticsearch@2ea9707867
Ensures that parent task is propagated to child operations to ensure that reindex operation can be cancelled if needed.
Original commit: elastic/x-pack-elasticsearch@fa40b5a951
This change fixes the creation of the bind DN string for active directory realms so that they do
not add the `@domain` suffix to the bind DN, when it is a actual DN value.
Original commit: elastic/x-pack-elasticsearch@bd04c07e16
This adds the Cloud detection mappings so that phone home can take advantage of their existence.
It also sets the system load fields to use `half_floats`.
Original commit: elastic/x-pack-elasticsearch@75f7992d38
This stops checking for older alerts now that we support emailing based on state changes. This only applies to 6.0 because the upgrade _can_ still happen usefully in 5.6 without the noise.
Original commit: elastic/x-pack-elasticsearch@9d73c64daa
The active directory user principal name format typically takes the form user@domain, which is what
the current implementation expects. However, active directory also allows the definition of other
suffixes that are not actual domains. A user can still authenticate using this user principal name
but the behavior of our realm would cause it to fail as it parsed the suffix as a domain and used it
as the search base for the user. Instead, we should use the default user search base and only look
for entries that have this exact user principal name. In a scenario where a realm is configured for
multiple domains in the same forest, the search base should be the base for the entire forest.
relates elastic/x-pack-elasticsearch#1744
Original commit: elastic/x-pack-elasticsearch@de00c4817e
Optional restrictions can be applied on top of an existing certificate trust scheme (PEM CAs, JKS TrustStore etc).
The restrictions reduce the set of certificate that would be otherwise trusted.
In this commit, the only supported restrictions are to filter by the certificate's SubjectAlternativeNames that are tagged as "other name" with an object-identifier of "cn"
Original commit: elastic/x-pack-elasticsearch@c6105a47df
This allows for messages to be returned, and distinguishes between 4 different results:
- I have authenticated the user
- I don't know how to authenticate that user. Try another realm.
- I tried to authenticate the user, but failed. Try another realm.
- I tried to authenticate the user, but failed. Fail the authentication attempt.
Original commit: elastic/x-pack-elasticsearch@f796949cfb
This commit adds support for a bind user when using the active directory realm. The addition of a
bind user also enables support for the user lookup mechanism, which is necessary to support the run
as functionality that we provide.
relates elastic/x-pack-elasticsearch#179
Original commit: elastic/x-pack-elasticsearch@40b07b3422
When there are data or master nodes in the cluster, that are older
than ES 6.0 alpha3, then watcher will only start on the master node.
Changed all transport actions to be master node actions, as there is
already a method to decide to run locally, which we can piggyback on.
Original commit: elastic/x-pack-elasticsearch@65cecb6d69
* [Monitoring] Update template version to 6000003
* update LAST_UPDATED_VERSION
* use the 6.0.0-beta1 version constant
* use the 6.0.0-beta1 version value (6000026)
* fix cluster uuid reference in terms filters
Original commit: elastic/x-pack-elasticsearch@40c822d396
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.
Original commit: elastic/x-pack-elasticsearch@926a7b2d86
In case of mixed version in a cluster the update could potentially downgrade the mappings. This CL changes the check to avoid this situation.
Original commit: elastic/x-pack-elasticsearch@9ecd5df70e
This commit provides an explicit name for the log tail thread, otherwise
the thread shows up with a generic name like Thread-12 in stack
dumps. While the stack trace provides enough information to conclude
what this thread is doing, it is better to be more explicit about the
purpose of this thread so that that can be discerned directly from the
name of the thread.
Additionally, we mark this thread as a daemon thread. Since this thread
is created by the main thread during node initialization it defaults to
being a user thread. Since the JVM only exits when the only threads
running are not user threads, if this thread were to somehow block then
it could prevent the server JVM from exiting. As such, this thread
should be marked as a daemon thread.
Relates elastic/x-pack-elasticsearch#1969
Original commit: elastic/x-pack-elasticsearch@b3761c7d91
This commit adds a setting to allow changing the user search filter. Previously the filter was a
simple equality filter that mapped a given attribute to the value of the username. The default
behavior remains the same with this change but provides additional flexibility to users to who may
need more advanced LDAP searches. The user attribute setting has been deprecated due to the overlap
with the new filter setting.
relates elastic/x-pack-elasticsearch#1861
Original commit: elastic/x-pack-elasticsearch@e9d797e81c
When having a mixed cluster with 5.6 and 6.0 nodes, the template upgrade
service has a cluster state listener that deletes the old watches and
triggered_watches index template. However during that time the 5.6 node
WatcherIndexTemplateRegistry checks if the templates are missing and
adds them back. This results in a race, because the new .watches index
template does not get added by the WatcherIndexTemplateRegistry when the
6.0 node is node a master node.
This commit circumvents this issue, by only deleting the watches and
triggered watches template during the upgrade process.
Original commit: elastic/x-pack-elasticsearch@71380f460a
Following elastic/elasticsearch#23997 this was only working due to
the way we were suppressing certain errors during job deletion.
This PR makes the situations we want to ignore during job deletion
clearer and adheres to the intention of elastic/elasticsearch#23997
by only specifying concrete indices to the `indices` arguments of
deletion calls.
Original commit: elastic/x-pack-elasticsearch@2458c3db40
If the cluster is still making cluster state updates while disabling the license, cluster state updates might not go through,
triggering an assertion failure at the end of the test that checks if all cluster states have been applied.
relates elastic/x-pack-elasticsearch#1627
Original commit: elastic/x-pack-elasticsearch@e11863fd02
Adds a update procedure for ML index mappings in order to allow adding new fields. The ES version is stored in the "_meta" field of the ML mappings which then get applied to every index. When opening a job, this version is checked on the (shared) index and a mapping update is performed in case that the version is older than current.
Original commit: elastic/x-pack-elasticsearch@211608c7ad
* add index checks
The following checks apply to both 5.x as well as 6.0
index checks:
- coercion of boolean fields
- the _all meta field is now disabled by default
- the include_in_all mapping parameter is now disallowed
- unrecognized match_mapping_type options not silently ignored
- similarity settings
- store settings
- store throttling settings
- shadow replicas have been removed
Original commit: elastic/x-pack-elasticsearch@5583b21763
This is related to elastic/x-pack-elasticsearch#1217. This commit adds a ClusterStateListener at
node startup. Once the cluster and security index are ready, this
listener will attempt to set the elastic user's password with the
bootstrap password pulled from the keystore. If the password is not in
the keystore or the elastic password has already been set, nothing will
be done.
Original commit: elastic/x-pack-elasticsearch@7fc4943c45
As the TemplateUpgradeService requires permissions to add and
delete index template, we have to grant those to the _system user.
This commit adds such permissions plus an integration test.
Original commit: elastic/x-pack-elasticsearch@a76ca9c738
The monitoring watches are roughly executing the same queries even when
they run against different clusters. However the way they were created,
where the cluster name is replaced via search & replace instead of using
watch metadata implies, that every watch is different from a script
compilation cache perspective. On top of that every of those watches is
executed once a minute. So if a new node becomes master and you monitor
three clusters, this results in a fair share of compilations in the first minute. The
reason for the compilation is the fact, that the search input uses
mustache for being able to add dynamic parts into a search using
mustache.
Several of those watches also need to compile more than one search
request.
The maximum default value for script compilations is only 15 and thus at
least one watch will not be executed due to failing script compilations.
This commit changes the four watches, so that the search requests are
cacheable. This means, no matter how many clusters you monitor, there
will be only needed four compilations for the different watches and
that's it.
Relates elastic/support-dev-help#2090
Original commit: elastic/x-pack-elasticsearch@6c877421bb
This test creates watches in old versions of elasticsearch, upgrades them after upgrading cluster to the latest version and then tests that they were upgraded correctly.
Original commit: elastic/x-pack-elasticsearch@b9d45eb2a5
* [Monitoring] Email actions for Cluster Alerts
* fix quotations in email fields
* move email vars to transform, and rename for snake_case
* add state to email subject for cluster status alert
* remove types field in kibana_settings search
* simplify email action condition script
* uppercase the state for the email subject
* only append state to email subject if alert is new
* show state in email subject even when alert is resolved
Original commit: elastic/x-pack-elasticsearch@e6fdd8d620
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.
This commit loosens two aspects of the security.
1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.
Original commit: elastic/x-pack-elasticsearch@8a2a577038
This is step 2 of elastic/x-pack-elasticsearch#1604
This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.
Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
ML has two types of custom cluster state:
1. jobs
2. datafeeds
These need to be parsed from JSON in two situations:
1. Create/update of the job/datafeed
2. Restoring cluster state on startup
Previously we used exactly the same parser in both situations, but this
severely limits our ability to add new features. This is because the
parser was very strict. This was good when accepting create/update
requests from users, but when restoring cluster state from disk it meant
that we could not add new fields, as that would prevent reloading in
mixed version clusters.
This commit introduces a second parser that tolerates unknown fields for
each object that is stored in cluster state. Then we use this more
tolerant parser when parsing cluster state, but still use the strict
parser when parsing REST requests.
relates elastic/x-pack-elasticsearch#1732
Original commit: elastic/x-pack-elasticsearch@754e51d1ec
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/24437
It changes two things, for the disable tests, it uses a valid endpoint instead
of a previously invalid endpoint that happened to return a 400 because the
endpoint was bad, regardless of if watcher was disabled.
The other change is to create the watches index by putting a watch using the
correct API, rather than manually creating the index. This is because
`RestHijackOperationAction` hijacks operations like this and stops accessing the
endpoint in a regular manner.
Original commit: elastic/x-pack-elasticsearch@3be78d9aea
Currently, the autodetect process has an `ignoreDowntime`
parameter which, when set to true, results to time being
skipped over to the end of the bucket of the first data
point received. After that, skipping time requires closing
and opening the job. With regard to datafeeds, this does not
work well with real-time requests which use the advance-time
API in order to ensure results are created for data gaps.
This commit improves this functionality by making it more
flexible and less ambiguous.
- flush API now supports skip_time parameter which
sends a control message to the autodetect process
telling it to skip time to a given value
- the flush API now also returns the last_finalized_bucket_end
time which allows clients to resume data searches correctly
- the datafeed start API issues a skip_time request when the
given start time is after the resume point. It then resumes
the search from the last_finalized_bucket_end time.
relates elastic/x-pack-elasticsearch#1913
Original commit: elastic/x-pack-elasticsearch@caa5fe8016
This fixes `testDeleteJobAfterMissingAliases` to not fail randomly.
The reason the test was failing is that at some point some aliases
are deleted and the cat-aliases API is called to verify they were
indeed deleted. This was checked by asserting an
index_not_found_exception was thrown by the cat-aliases request.
This was some times working as there were no other aliases. However,
that depends on whether other x-pack features had time to create their
infrastructure. For example, security creates an alias. When other
aliases had the time to be created, the cat-aliases request does not
fail and the test fails.
This commit simply changes the verification that the read/write
aliases were deleted by replacing the cat-aliases request with
two single get-alias requests.
Original commit: elastic/x-pack-elasticsearch@fe2c7b0cb4
The logging shows a wrong HTTP response status code from a previous
request. In addition the body now also gets logged, as debugging
is impossible otherwise.
Original commit: elastic/x-pack-elasticsearch@cc998cd587
This changes from collecting every index statistic to only what we actually want. This should help to reduce the performance impact of the lookup.
Original commit: elastic/x-pack-elasticsearch@80ae20f382
This is related to elastic/x-pack-elasticsearch#1217 and elastic/x-pack-elasticsearch#1896. Right now we are checking if an
incoming address is the loopback address or a special local addres. It
appears that we also need to check if that address is bound to a
network interface to be thorough in our localhost check.
This change mimicks how we check if localhost in `PatternRule`.
Original commit: elastic/x-pack-elasticsearch@a8947d6174
Depending on the random numbers fed to the analytics,
it is possible that the first planted anomaly ends up
in a different bucket due to the overlapping buckets feature.
Then that may result to a single interim bucket being available
due to overlapping buckets blocking the other interim bucket
from being considered.
I am removing the initial anomaly from the test as it is not useful
and it makes the test unstable.
relates elastic/x-pack-elasticsearch#1897
Original commit: elastic/x-pack-elasticsearch@aca7870708
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.
Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.
This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.
Original commit: elastic/x-pack-elasticsearch@e1e101a237
We made the mistake to generate way to many settings in xpack which makes
finding out the right string and where it's defined super difficult. If
we use constants we can just use commandline tools to find where the settings
are defined. This also removes 1.x and 2.x BWC from the enabled settings which should
be removed in 6.x
Original commit: elastic/x-pack-elasticsearch@ec25e6c40c
This is related to elastic/x-pack-elasticsearch#1217. This commit reads two environment variables on
startup: BOOTSTRAP_PWD and ELASTIC_CONTAINER. If BOOTSTRAP_PWD is
present, ELASTIC_CONTAINER must be set to true. Otherwise a new
bootstrap check will fail.
If ELASTIC_CONTAINER is set to true, the elastic user can be
authenticated with the BOOTSTRAP_PWD variable when its password
has not been explicitly set.
Original commit: elastic/x-pack-elasticsearch@78f53fd232
The logic of resetting acknowledgements is only executed, if the watch
wide condition is not met. However, if you dont specify a condition
(which makes it always true), but create a condition in your action
(this might make sense because it allows you to execute a transform and
then execute the condition), then after acking this action, it will
never get be unacked, because the watch wide condition is always met.
relates elastic/x-pack-elasticsearch#1857
Original commit: elastic/x-pack-elasticsearch@95aa402c27
The .security index used several different types to differentiate the
documents added to the index (users, reserved-users, roles, etc). Since
types are deprecated in 6.x, this commit changes the .security index
access layer to only use a single type and have all documents in the
index be of that single type. To differentiate documents that may have
the same id (e.g. the same user name and role name), the appropriate
type of the document is prepended to the id. For example, a user named
"jdoe" will now have the document id "user-jdoe".
This commit also ensures that any native realm security index operations
that lead to auto creation of the security index first go through the process
of creating the internal security index (.security-v6) and creating the alias
.security to point to the internal index.
Lastly, anytime the security index is accessed without having been
upgraded, an exception is thrown notifying the user to use the
upgrade API to upgrade the security index.
Original commit: elastic/x-pack-elasticsearch@cc0a474aed
Adds a new Upgrade API with the first action, index upgrade info, that returns that list of indices that require upgrade in the current cluster before the cluster can be upgraded to the next major version.
Relates to elastic/x-pack-elasticsearch#1214
Original commit: elastic/x-pack-elasticsearch@761e7d2128
* [Logstash] Change management license to Gold
Previously the license type for LS config management was `BASIC`. In order to use the security features in Standard/Gold, we had to bump Logstash as well to Gold license.
relates elastic/x-pack-elasticsearch#1841
Original commit: elastic/x-pack-elasticsearch@29194b2417
Adds REST endpoint and Transport Action for retrieving breaking-changes deprecations that exist in current version. This PR is just the framework for such an API, future checks will be added to the appropriate branches.
Original commit: elastic/x-pack-elasticsearch@990e3468e9
This commit adds new settings for the ssl keystore (not the ES keystore)
passphrase settings. New setting names are used, instead of trying to
support the existing names in both yml and the ES keystore, so that
there does not need to be complicated logic between the two. Note that
the old settings remain the only way to set the ssl passphrases for the
transport client, but the Settings object for transport clients are
created in memory by users, so they are already as "secure" as having a
loaded ES keystore. Also note that in the long term future (6.x
timeframe?) these settings should be deprecated and the keys/certs
themselves should be moved into the ES keystore, so there will be no
need for separate keystores/passphrases.
relates elastic/elasticsearch#22475
Original commit: elastic/x-pack-elasticsearch@be5275fa3d
* The TriggeredWatchStore now only has one method to put triggered
watches
* All code is async in TriggeredWatchStore, locking has been removed
* The dedicated WatchRecord.Fields interface has been removed
* TriggeredWatchTests integration test has been moved to a unit test
Original commit: elastic/x-pack-elasticsearch@bc4b5820fb
`index.mapper.single_type` will be removed in master. While there is still
one usage in the security template that we are working on, this change
will remove the remaining usage.
Original commit: elastic/x-pack-elasticsearch@6e7f63b9e0
This is just a workaround at the moment, but allows to use
mustache if you only provide the `url` part of a request,
instead of scheme, port, path, host, etc.
Original commit: elastic/x-pack-elasticsearch@3a4aa26665
Every cluster state update resulted in a log message, that watcher
pauses execution. This has been fixed to only log, if there was an
actual state switch from executing to pausing, but do nothing if
there are no local shards anyway.
This will reduce the logging noise in tests a lot.
Original commit: elastic/x-pack-elasticsearch@32ab86610c
* Give kill a chance to kill the process before closing input
* Remove variable that can be refactored out
Original commit: elastic/x-pack-elasticsearch@42f7a3cece
The graph API needs to be able to search in remote indices. Although it uses the Search API to perform the search and so doesn’t need to deal with remote indexes directly, the security feature needs to know it can be used with remote indexes so it knows to include remote indices in the list of indices accessible from the API for index level security
Original commit: elastic/x-pack-elasticsearch@e3cd84963e
This change removes all local security checks against remote cluster names.
Any user is allowed to attempt a cross-cluster search, and it is the responsibility of the remote cluster to authorise the search (or not).
This includes support for remote searches even if you have _no_ local search privileges.
Original commit: elastic/x-pack-elasticsearch@1620c3a8fa
Currently, aggregated datafeeds construct JSON from the aggregation
response by traversing all nested aggregations. In order to
achieve this, multiple leaf aggregations are not supported. Also,
scenarios it makes it impossible to effectively use pipeline
aggregations as it will not ignore the intermediate bucket
aggregations.
This commit refactors AggregationToJsonProcessor in order to
support the above scenarios. This is achieved by only converting
the fields of interest, that is the job analysis fields.
Original commit: elastic/x-pack-elasticsearch@8b575956ca
This changes the validation criteria we use for user and role
names in the file realm, native realm, and the
realm-agnostic code in x-pack security. The new criteria is:
A valid username's length must be at least 1 and no more than 1024
characters. It may not contain leading or trailing whitespace. All
characters in the name must be be alphanumeric (`a-z`, `A-Z`, `0-9`),
printable punctuation or symbols in the https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block)[Basic Latin (ASCII) block],
or the space character.
Original commit: elastic/x-pack-elasticsearch@f77640f269
Today we have some hidden complexity related to default configurations
might specify NO_KEY which is in some cases valid for server configuration.
This change removes the leniencey paramenters on the validation methods and removes
obsolet asserts.
Original commit: elastic/x-pack-elasticsearch@17ed4b1d20
You can now disable cluster alerts in the local exporter, which you can do in the HTTP exporter already.
This helps users that mess up their watcher configuration (e.g., disabling scripts) can turn off the feature to avoid log spam.
Original commit: elastic/x-pack-elasticsearch@f2096b553d
The cluster alert did not get updated when we dropped the logical 'type' for cluster_state in favor of merging it with cluster_stats in 5.5+.
Original commit: elastic/x-pack-elasticsearch@c7105be36f
Today we parse / construct SSLConfiguration late when client / server
channels are created. This is problematic if we try to read from secure settings
in the future since we need to read all secure settings as soon as the node is
constructed. If we keep on reading late, we will access a closed keystore
since channel creation happens during node startup.
Original commit: elastic/x-pack-elasticsearch@78d6061990
If the internal index version of an index is not the right one, do
not start watcher.
Also, add the internal index version of 6 to all our index templates.
Original commit: elastic/x-pack-elasticsearch@20b50aa82b
If multiple job deletion requests were sent in quick succession, there was a
race condition that meant they could both get through the check to enforce
one active deletion request at a time. Then, if the job was immediately
recreated after the first deletion request returned, the second, still running,
deletion request could interfere with it and delete the aliases that the put
job request created.
This problem can be avoided by using the "ask forgiveness, not permission"
idiom when checking if the job is already being deleted at the beginning of
each deletion request.
Additionally, now even force delete requests will wait for a certain amount
of time for a prior delete request to complete. This is to avoid the same
race conditions. However, force delete requests will eventually start an
(unsafe) parallel delete to provide a get-out in case a delete request
completely dies.
relates elastic/x-pack-elasticsearch#1765
Original commit: elastic/x-pack-elasticsearch@b5c8f26a0e
The current testing setup only checked if watcher was started, but it
also needs to check for the index template in order to be sure that
everything is set up correctly, before trying to put a watch.
relates elastic/x-pack-elasticsearch#1762
Original commit: elastic/x-pack-elasticsearch@3ed78b15a1
This commit changes a couple of places where our ExceptionsHelper
class was throwing exceptions to instead return the exceptions.
Then they can be passed to onFailure() methods or thrown depending
on what's appropriate for the caller. This is the standard Elastic
way of handling failures.
Original commit: elastic/x-pack-elasticsearch@fce07eb075
This allows to configure a proxy for the reporting attachment
action. The proxy is used by the HTTP client.
Original commit: elastic/x-pack-elasticsearch@87b6ab1b68
The MockTerminal used in tests uses \n always and the reverted commit re-introduced the bug which
had been fixed earlier.
Original commit: elastic/x-pack-elasticsearch@09b93b5565
This change enables closing a job while it is in
the middle of restoring its state. This is has the
benefit of allowing users to close jobs that due to
relocation are `opened` but they are still restoring
state. It also helps avoiding race conditions in tests.
Part of this change also includes restoring the state
as a separate step from the process creation. This means
we no longer block the job map while the process is
restoring its state.
relates elastic/x-pack-elasticsearch#1270
Original commit: elastic/x-pack-elasticsearch@1713a4a7c4
In cases where the job is bound on the analytics performance
the datafeed can fail because the scroll expires. This is
commit increases the scroll context duration from 10 to 30 minutes
as a temporary solution that will avoid most cases.
Original commit: elastic/x-pack-elasticsearch@fd277bbaa1
This changes the native realm migrate tool tests to use the System.lineSeperator instead of `\n`
so that the tests will pass on Windows.
Original commit: elastic/x-pack-elasticsearch@d3f9a71ac4
Prior to this change, if the persistent tasks framework noticed that a
job was running on a node that was isolated but has rejoined the cluster
then it would close that job. This was not ideal, because then the job
would persist state from the autodetect process that was isolated. This
commit changes the behaviour to kill the autodetect process associated
with such a job, so that it does not interfere with the autodetect process
that is running on the node where the persistent tasks framework thinks it
should be running.
In order to achieve this a change has also been made to the behaviour of
force-close. Previously this would result in the autodetect process being
gracefully shut down asynchronously to the force-close request. However,
the mechanism by which this happened was the same as the mechanism for
cancelling tasks that end up running on more than one node due to nodes
becoming isolated from the cluster. Therefore, force-close now also kills
the autodetect process rather than gracefully stopping it. The documentation
has been changed to reflect this. It should not be a problem as force-close
is supposed to be a last resort for when normal close fails.
relates elastic/x-pack-elasticsearch#1186
Original commit: elastic/x-pack-elasticsearch@578c944371
This came up in a forum post. An NPE was raised, when a search input
contained a search that did not contain a body, but just specified
indices or types.
This commit allows for empty bodies, and also makes sure there are
no null pointer exceptions by using empty bytes references otherwise.
In addition a suite scoped integration test was converted to a unit
test.
Original commit: elastic/x-pack-elasticsearch@29be2976fc
This introduces a new index setting called xpack.internal.format to
x-pack, which is configured for all of our index templates and set to
"v6". This indicates the version of compatibility of this index.
In addition a setting named index.xpack.version has been removed,
as it was unused.
Watcher does not start, if the watches and the triggered watches
index is not compatible with this setting.
Original commit: elastic/x-pack-elasticsearch@e430691c51
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
Original commit: elastic/x-pack-elasticsearch@d01b0df1d9
Before the event stats were mislabeled, so were not being indexed, and
the ephemeral_id was only in the _state document.
Original commit: elastic/x-pack-elasticsearch@ca0ec81aa5
Adds tests similar to `:qa:full-cluster-restart` for x-pack. You
run them with `gradle :x-pack:qa:full-cluster-restart:check`.
The actual tests are as basic as it gets: create a doc and load it,
shut down, upgrade to master, startup, and load it. Create a user
and load it, shut down, upgrade to master, startup, and load it.
Relates to elastic/x-pack-elasticsearch#1629
Original commit: elastic/x-pack-elasticsearch@8994bec8e7
This commit removes unnecessary initialization of the system key in tests that no longer make use
of the system key. It also removes the feature usage for the system key in the SecurityFeatureSet.
Original commit: elastic/x-pack-elasticsearch@b9fffe0bd3
Logstash now has ephemeral id at the instance level and also at the
pipeline level, we need to add them to the logstash monitoring template.
Original commit: elastic/x-pack-elasticsearch@dfac702d59
* Use bulk request to persist model plots and model size stats
* Revert persisting model size stats in the bulk request
* Refactor results persister
Original commit: elastic/x-pack-elasticsearch@f51297bfc2
We recently added logging for critical authentication failures as they had previously been silent (with respect to logs) but would cause authentication processing to stop.
However, the reserved realm intentionally uses exceptions to stop any other realm authenticating a reserved user if the password is entered incorrectly.
Since this is the most common use of exceptions in the authc chain, we reduce the logging verbosity in normal cases (drop the stack trace, remove "unexpected") and only log the full details in debug.
Original commit: elastic/x-pack-elasticsearch@686a98010b
This is related to elastic/x-pack-elasticsearch#1217. This change introduces a tool
bin/x-pack/setup-passwords that will streamline the setting of
internal user passwords. There are two modes of operation. One mode
called auto, automatically generates passwords and prints them to
the console. The second mode called interactive allows the user to
enter passwords.
All passwords are changed using the elastic superuser. The elastic
password is the first password to be set.
Original commit: elastic/x-pack-elasticsearch@00974234a2
After improving the authorization of scroll requests and backporting to 5.x, we no longer need to
have any signing code in master. This commit removes it.
Original commit: elastic/x-pack-elasticsearch@8b65fd9338
If an exception occurs while sending the initial setup messages to the autodetect
such that it fails rather than reaching the open state then the autodetect process
needs to be killed to prevent it hogging resources.
Relates elastic/x-pack-elasticsearch#1684
Original commit: elastic/x-pack-elasticsearch@1ee80ed9b0
In does not make sense for the time_field in the data_description to
be used as a by/over/partition field name, nor the summary_count_field,
categorization_field or as an influencer. Therefore, configurations
where the time_field in the data_description is used in the
analysis_config are now rejected.
Additionally, it causes a problem communicating with the C++ code if
the control field name (which is '.') is used in the analysis_config,
so this is also rejected at the validation stage.
Relates elastic/x-pack-elasticsearch#1684
Original commit: elastic/x-pack-elasticsearch@e6750a2cda
- Don't attempt to upgrade from 2.x
- Attempt up to 10 retries if the migration fails (with increasing back-off between attempts)
- If a cached user is disabled, recheck with the underlying store
The last change is required if the migration takes a long time.
While users are being migrated, they might be marked as disabled, but when the migration is complete they need to be usable immediately.
Original commit: elastic/x-pack-elasticsearch@2621867014
Removes the `assemble` task from projects that aren't published
to speed up `gradle assemble` so the unified release can call it.
Original commit: elastic/x-pack-elasticsearch@43dfcc15f3
These tests are starting their own nodes and do not use the testing
trigger schedule class.
There are occasional test failure due to a race condition where watcher
is in the process of being started, but cannot be shut down properly,
because starting up was not finished when the shut down was called for.
These filter tests do not rely on watcher, so we can disable them for
now, but we still need to fix a race condition in starting/stopping
watcher.
relates elastic/x-pack-elasticsearch#1422
Original commit: elastic/x-pack-elasticsearch@f13bb7a6fb
There is no need to handle any _status field in
the 6.0 release from now on, as everything has been
taken care in the upgrade API.
Original commit: elastic/x-pack-elasticsearch@606581f4d7
This changes part of the logic that was added in elastic/x-pack-elasticsearch#644 and extended
in elastic/x-pack-elasticsearch#1495 so that when ML is disabled we never try to communicate with
the native controller during node shutdown.
The original reason for needing to communicate with the native controller
when ML is disabled was the problem of elastic/prelert-legacy#803.
However, this was fixed in a better way in elastic/elasticsearch#24579.
Now there is considerable benefit in never talking to the native
controller from the plugin code when ML is disabled, because it means
anyone suffering some obscure problem with ML can disable it without
running the risk of uncovering some other obscure problem with shutdown.
Original commit: elastic/x-pack-elasticsearch@9d329483a7
This adds a check in the REST tests to ensure that
watcher is started, and if not, tries to start watcher.
This eliminates test failures where watcher was not in
the correct state due to other tests stopping watcher.
Original commit: elastic/x-pack-elasticsearch@fc547d49b4
This commit adds a new Logstash component to x-pack to support the config management work. Currently, the functionality in this component is really simple; all it does is upload a new index template for `.logstash` index. This index stores the actual LS configuration.
On this template is bootstrapped in ES, Kibana can write user-created LS configs which adhere to the mapping defined here. In the future, we're looking into adding more functionality on the ES side to handle config documents, but for now, this is simple.
relates elastic/x-pack-elasticsearch#1499, relates elastic/x-pack-elasticsearch#1471
Original commit: elastic/x-pack-elasticsearch@d7cc8675f7
In the case where a field is a text multi-field, it has
no doc values and it is not in source. Thus, the datafeed
will not be able to extract it.
However, it is possible to extract it by getting its parent
field instead. This commit implements the logic to look
in parent fields when the field in question is a text field.
Original commit: elastic/x-pack-elasticsearch@f116e89921
In 5.4.x, the datafeed attempts to get all fields from
doc_values by default. It has a `_source` parameter which
when enabled changes the strategy to instead try to get
all fields from the source.
This has been the most common issue users have been
reporting as it means the datafeed will fail to fetch
any text fields by default.
This change uses the field capabilities API in order
to automatically detect whether a field is aggregatable.
It then extracts such fields from doc_values while the
rest are taken from source. The change also adds
validation to the start datafeed action so that if
fields are missing mappings or the time field is not
aggregatable we respond with an appropriate error.
relates elastic/x-pack-elasticsearch#1649
Original commit: elastic/x-pack-elasticsearch@76e2cc6cb2
Specifying types for a datafeed should be optional
as no types is equal to searching through all types.
Original commit: elastic/x-pack-elasticsearch@f61ac01b45
Only log an entry when an actual stop is executed instead of
always logging.
Also added a reason to stop watcher to the methods, so that
debug logs will yield that information.
Original commit: elastic/x-pack-elasticsearch@8efaed0e9a
The constructor for CreateIndexResponse changed to include the index
name. This commit adapts x-pack-elasticsearch to this change.
Original commit: elastic/x-pack-elasticsearch@b078d80cd9
Commit elastic/x-pack-elasticsearch@b07aa78a7b was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement and project name.
Original commit: elastic/x-pack-elasticsearch@9a77269fa6
This is mainly a commit to reduce noise in test logfiles when going
through them. When watcher shuts down and another node takes over, it
might try to start watcher again and tries to load triggered watches.
However the triggered watches index could be gone in the meantime due to
further shutdown. This results in logging a stack trace that the index
does not exist.
This commit checks the cluster state before trying to load triggered
watches to prevent an IndexNotFoundException in the logs.
Original commit: elastic/x-pack-elasticsearch@9f26d557d0
This introduces a check to only pause the execution of watcher
when there is no metadata but there was a shard on this node before
that inside of the ClusterStateListener.
This prevents repeated logging that watcher was paused even
though it was not necessary to call anything.
Original commit: elastic/x-pack-elasticsearch@8d3a829ffb
Watcher had an undocumented feature to configure the settings of any
index template via updating the cluster settings. Instead of changing
the template one could add a setting during runtime that overwrote
what is written in the index template.
As index template are created once and not overwritten, users should
just change the index template - or the concrete index settings like
number of replicas.
This feature was not exposed in our documentation at all.
Original commit: elastic/x-pack-elasticsearch@32e1769925
This removes the Cluster State collector and resolver and moves the collection of the cluster state (and cluster health, which is already included in cluster stats).
This makes the tests a little more stable and removes an extra network hop during monitoring data collection.
Original commit: elastic/x-pack-elasticsearch@44851d2dd6
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
Original commit: elastic/x-pack-elasticsearch@b07aa78a7b
The aim of this change is to prevent many identical requests to create
watcher index templates being submitted when a cluster first starts up
and many cluster state updates are happening. Prior to this change, if
watcher's original index template creation requests queued up behind other
cluster state change requests then for each other request watcher would
re-request creation of all its index templates. After this change it
uses a strategy similar to that used by ML to only have one creation
request per index template in the cluster state change queue at any time.
Relates elastic/x-pack-elasticsearch#1368
Relates elastic/x-pack-elasticsearch#1631
Relates elastic/x-pack-elasticsearch#1650
Original commit: elastic/x-pack-elasticsearch@ad87bf3f78
Although the job stats for jobs with missing results indices are clearly
ruined, it's better to provide zeroes for the missing values and show the
stats for other jobs than to fail the whole request. This means the UI
can continue to function.
relates elastic/x-pack-elasticsearch#1656
Original commit: elastic/x-pack-elasticsearch@a06fa994a5
* Wait for job deletion if it is in the deleting state
* Tolerate errors if multiple force delete requests
Original commit: elastic/x-pack-elasticsearch@1f0c9fbb86
When a user executes a watch and specifies it as part of the
execute watch API, no triggered watch is created, as the watch
cannot be picked up anymore (it only leaves for the duration of
the request).
However until now the TriggeredWatchStore was invoked and tried
to delete this non-existing triggered watch, resulting in some
log cluttering.
This commit removes this try to delete a non-existing triggered
watch.
Original commit: elastic/x-pack-elasticsearch@3db125cea2
This is just the culmination of all of the minor PRs associated with 1068. It will:
- Drop the `.monitoring-data-N` index
- Drop use of `_type` in all cases (replaced by `doc` and a new `type` field)
- Drop the API version from the template name (e.g., instead of `.monitoring-es-6` we now use `.monitoring-es`).
- Change API version to `-6-` from `-2-`.
- Both exporters handle versioned resources (templates, pipelines, and watches)
- HTTP exporters will optionally (true by default) publish placeholders for the old, `-2` templates.
When this is backported, it will need to:
- Change `index_patterns` to `template` within the templates.
- Downgrade the version requirements for the templates, pipeline, and watches _and_ the HTTP exporter itself (all require 6.0)
This is a companion to the feature branch in X-Pack Kibana elastic/x-pack-kibana/pull/1318 and they need to be merged at the same time.
Original commit: elastic/x-pack-elasticsearch@6031cfffa4
This commit adds better security for scroll requests in that they are now tied to a single user as
we only authorize the request that creates the scroll. This is accomplished by adding a
SearchOperationListener that listens for new scroll contexts and stores the authentication on the
ScrollContext. Then upon
retrieval of the search context for a query or fetch, the current authentication is compared to the
authentication that was present when the scroll context was created. If the current authentication
belongs to a different user, then a SearchContextMissingException will be thrown to prevent leaking
a valid vs invalid scroll id.
Additionally, signing of a scroll id is only performed when there is a older node in the cluster
that would expect the scroll id to be signed. Once this is backported to 5.x, we can remove this
bwc layer for 6.0/master.
Original commit: elastic/x-pack-elasticsearch@0e5dcafd32
When a user or client intend to delete a datafeed
and its job, there is benefit into ensuring the
datafeed has gracefully stopped (ie no data loss).
In constrast, the desired behaviour is to stop and
delete the datafeed as quickly as possible.
This change adds a force option to the delete
datafeed action. When the delete is forced,
the datafeed is isolated, its task removed and,
finally, the datafeed itself is removed from the
metadata.
relates elastic/x-pack-elasticsearch#1533
Original commit: elastic/x-pack-elasticsearch@5ae0168bf2
We try to install empty ML metadata as soon as possible after startup
if none exists. However, this still leaves a short gap when the cluster
is active with no ML metadata. To avoid problems, functions that use
the ML metadata should treat this situation as equivalent to having
empty ML metadata.
relates elastic/x-pack-elasticsearch#1643
Original commit: elastic/x-pack-elasticsearch@8f0e00cda8
ignoreAliases allows to resolve index expressions against concrete indices only, rather than against indices and aliases. It is used for now only in IndicesAliasesRequest and the indices resolution code in the security plugin needs to be adapted accordingly.
Original commit: elastic/x-pack-elasticsearch@ae964eade9
This commit switches over to two index aliases per job: one for reading
and one for writing. In the future this will allow the addition of a
rollover endpoint for ML results indices. (Rollover is still not possible
following this change, but the change to make it possible in the future
should not be a breaking change now.)
Relates elastic/x-pack-elasticsearch#1599
relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@d648f4631f
* Add force delete job option
* Can’t kill a process on a 5.4 node
* Address review comments
* Rename KillAutodetectAction -> KillProcessAction
* Review comments
* Cancelling task is superfluous after it has been killed
* Update docs
* Revert "Cancelling task is superfluous after it has been killed"
This reverts commit 576950e2e1ee095b38174d8b71de353c082ae953.
* Remove unnecessary TODOs and logic that doesn't alwasys force close
Original commit: elastic/x-pack-elasticsearch@f8c8b38217
This test would sometime leak threads.
The "Timer thread for LDAPConnection" is created by the unboundid SDK - closing the connection should force the thread to terminate
Original commit: elastic/x-pack-elasticsearch@bd58a17a59
The cluster state listener to decide if watcher should be reloaded was
assuming that no aliases could be used and thus wrongly could trigger
a reload, which could have lead to wrong test results.
During debugging I also added a reason for reloading and fixed another
wrong test assumption.
Also the listener does not rely on previous cluster state, but stores this
in instance variable, as we need to compare with local state and not the
previous cluster state.
Original commit: elastic/x-pack-elasticsearch@582783a66d
Includes:
- Extensive changes to "mapping roles" section
- New section for role mapping API
- Updates to LDAP/AD/PKI realms to refer to API based role mapping
- Updates to LDAP/AD realms: `unmapped_groups_as_roles` only looks at file-based mappings
- Updates to LDAP/AD realms: new setting for "metadata"
Original commit: elastic/x-pack-elasticsearch@6349f665f5
This avoids log spam about being unable to create new mappings in indices
that are set to only allow one type. (It doesn't actually have any effect
on the deletion, which was working before despite the failure to create new
mappings for the legacy types referenced by the delete request.)
relates elastic/x-pack-elasticsearch#1634
Original commit: elastic/x-pack-elasticsearch@061ce7acf1
Because:
1. It's pointless, as new detector_index values are assigned when an
analysis_config is parsed
2. It creates a backwards compatibility issue when upgrading from v5.4
Original commit: elastic/x-pack-elasticsearch@2f61aa457e
This is the x-pack side of the removal of `accumulateExceptions()` for both `TransportNodesAction` and `TransportTasksAction`.
There are occasional, random failures that occur during API calls that are silently ignored from the caller's perspective, which also leads to weird API responses that have no response and also no errors, which is obviously untrue.
Original commit: elastic/x-pack-elasticsearch@9b57321549
Detectors now have a field called detector_index. This is also now the
field that needs to be supplied when updating a detector. (Previously
it was simply index, which was confusing.)
When detectors are added to an analysis_config it will reassign
ascending detector_index values starting from 0. The intention is
never to allow deletion of detectors from an analysis_config, but
possibly to allow disabling them in the future. This ensures that
detector_index values in results will always tie up with detector_ids
in the detectors that created them.
relates elastic/x-pack-elasticsearch#1275
Original commit: elastic/x-pack-elasticsearch@20a660b07b
At the end of the test, LocalExporterTests checks if no more monitoring
data are exporter by checking multiple times the last time nodes_stats
documents were exported, stopping after 10 seconds. It does this in a
@After annotated method but it would be better to do this in a finally
block. Also, it should search for node_stats documents only if the
monitoring indices exist and are searchable to avoid some "all shards
failed" failures.
Original commit: elastic/x-pack-elasticsearch@90ffb4affd
Now we've set the option for one type per index it causes a stack trace
in to be logged if we issue a request to delete two documents with
different types. We only do this to cover the case of documents left
over from v5.4. We can avoid it by deleting by query using just the
document IDs.
Original commit: elastic/x-pack-elasticsearch@2abffc7d95
This commit removes ClientProxy and WatcherClientProxy classes. They
were added in times, where there were issues with guice and circular
dependencies. However there is no guice anymore and on top of that
the classes do not add any value.
We can switch to use a regular client, but have to make sure that
the InternalClient is injected in all the transport actions as those
is able to query data, when security is enabled.
Original commit: elastic/x-pack-elasticsearch@763a79b2f7
The goal of this change is to allow datafeeds to start
when the job is in the opening state. This makes the API
more async and it allows clients like the ML UI to open a
job and start its datafeed without having to manage the
complexity of dealing with timeouts due to the job taking
time to open due to restoring a large state.
In order to achieve this, this commit does a number of things:
- accepts a start datafeed request when the job is opening
- adds logic to the DatafeedManager to wait before running the
datafeed task until the job is opened
- refactord the datafeed node selection logic into its own class
- splitd selection issues in critical and non-critical with regard
to creating the datafeed task
- refactord the unit tests to make simpler to write & understand
- adds unit tests for added and modified functionality
- changes the response when the datafeed cannot be started to
be a conflict exception
relates elastic/x-pack-elasticsearch#1535
Original commit: elastic/x-pack-elasticsearch@c83196155d
In preparation for the removal of types, new security types like invalidated-tokens are stored in the .security
index under the generic "doc" type, with a query filter on `doc_type`.
In order to avoid id clashes, we also need to use that doc_type as part of the document id.
relates elastic/x-pack-elasticsearch#1300
Original commit: elastic/x-pack-elasticsearch@469724a228
When an error response contains multiple layers of errors, Kibana displays
the one labelled root_cause. The definition of root_cause is the most
deeply nested ElasticsearchException. Therefore, it is of great benefit to
the UI if our config validation returns the actual problem in an
ElasticsearchException rather than an IllegalArgumentException.
This commit also adds an extra validation check to catch the case of a
single job config containing fields x.y as well as x earlier. Previously
this was caught when we tried to create results mappings, and was
accompanied by an error suggesting that using a dedicated results index
would help, when clearly it won't for a clash in a single job config.
Fixeselastic/x-pack-kibana#1387Fixeselastic/prelert-legacy#349
Original commit: elastic/x-pack-elasticsearch@7d1b7def6c
Otherwise it's possible that the get_filter endpoint can return a filter that's been
deleted. Although this is the behaviour of the search API, specific metadata
management APIs should provide better guarantees.
Original commit: elastic/x-pack-elasticsearch@818495f176
This allows us to build both 5.5.0-SNAPSHOT and 5.4.1-SNAPSHOT
artifacts for backwards compatibility testing. It is a port of
elastic/elasticsearch:24870 to x-pack and will be super useful
when elastic/elasticsearch:24846 is ported to x-pack.
Original commit: elastic/x-pack-elasticsearch@0ea443f488
Previously there were two @After methods in the XPackRestIT class, and
there is no guarantee about the order in which these run. This commit
replaces these with a single @After method that calls the cleanup methods
in a well-defined order.
Original commit: elastic/x-pack-elasticsearch@d3ab366591
The has_privileges API now supports wildcards.
The semantics are that the user must have a superset of the wildcard being checked.
---------------------
Role | Check | Result
---------------------
* | foo* | true
f* | foo* | true
foo* | foo* | true
foo* | foo? | true
foo? | foo? | true
foo? | foo* | false
foo | foo* | false
Original commit: elastic/x-pack-elasticsearch@817550db17
We don't hyphenate metadata anywhere else.
Also added tests for the LdapMetaDataResolver as they were completely absent.
Original commit: elastic/x-pack-elasticsearch@eec647ba93
This deletes tests getting results from MlJobIT since
such tests already exist in a form that is simpler to
understand and maintain in the YAML suite.
Original commit: elastic/x-pack-elasticsearch@b708e24877
This commit means that newly created ML state indices will have a single
type named "doc", and newly persisted state documents will have type
"doc" too.
Retrieving state is only supported for type "doc".
When deleting state, documents with the old types are deleted in addition
to those with type "doc". This means jobs created by the beta can be fully
deleted.
Relates elastic/x-pack-elasticsearch#668
Original commit: elastic/x-pack-elasticsearch@29c07d40f1
This commit adds an internal project call ml-cpp-snapshot which when
built will pull the ml cpp zip file from the prelert bucket. The GET
request has retries added to handle the dynamic aws creds eventual
consistency.
Original commit: elastic/x-pack-elasticsearch@1bba7d0f08
This commit cleans up the check for SSL with client authentication when a PKI realm is enabled by
moving it from the realm to a actual bootstrap check.
A bug was found during this cleanup in the check for transport profiles and that is also fixed in
this commit.
relates elastic/x-pack-elasticsearch#420
Original commit: elastic/x-pack-elasticsearch@3aa6a3edc0
The commit that converted the results index into single type
broke the search for fetching results for renormalization.
This commit fixes that.
Original commit: elastic/x-pack-elasticsearch@1ca7517adc
REST endpoints that support GET and POST need
to also support source parsing. As these
endpoints can accept a body but some clients
do not allow doing a GET with a request body,
elasticsearch has support for parsing via a
source URI parameter. This commit adds source
handling to all such endpoints.
relates elastic/x-pack-elasticsearch#1204
Original commit: elastic/x-pack-elasticsearch@3949ea31fe
Previously we used to normalize records with their buckets. This required
nested scrolls: an outer scroll over buckets, then a nested scroll for
records in each bucket. This was fragile.
The new approach is to simply scroll first through buckets, then through
records. This is made possible because we no longer store max_record_score
on buckets nor bucket anomaly_score on records.
While making these changes I noticed that the PerPartitionMaxProbabilities
class was redundant (because it was storing max_record_score in the case of
per-partition normalization), so I removed it. I also removed a redundant
Map from the Bucket class and fixed its equals() and hashCode() methods.
relates elastic/x-pack-elasticsearch#1115
Original commit: elastic/x-pack-elasticsearch@efbee63573
Deleting a job issues 2 cluster state updates.
The first marks the job as deleted.
The second actually removes the job.
Both check that there is no datafeed referring to the job.
If a put datafeed request arrives between those 2 cluster
state updates, the datafeed gets created and the final
job cluster state update fails. This means we end up with
both the job and the datafeed, but the job's results and
state have been deleted.
This commit changes the behaviour so that the put
datafeed request fails for a job that is marked as deleted
as this scenario is avoiding partially executing an action.
relates elastic/x-pack-elasticsearch#1510
Original commit: elastic/x-pack-elasticsearch@76fa0f0b1a
This commit changes all results to use the single doc type.
All searches were adjusted to work without the need to specify
type which ensures BWC with 5.4 results.
Additional work is needed to put the new doc mapping in indices
created with 5.4 but it will be done in separate PR.
Relates elastic/x-pack-elasticsearch#668
Original commit: elastic/x-pack-elasticsearch@041c88ac2d
* Reduced a longish timeout to a shorter one, as a watch should be
executed in a HTTP test.
* Ensured that the TimeThrottleIntegration tests only query for own
watches in the watch history, also use random names for watch ids
* HipChatServiceTests configured deprecated logging package, so it was
not possible to follow the HTTP calls to the hipchat service endpoint.
relates elastic/x-pack-elasticsearch#1514
Relates elastic/x-pack-elasticsearch#1515
Original commit: elastic/x-pack-elasticsearch@adb492e4e9
* Remove sequenceNum from anomaly records and influencers
* Generate unqiue IDs without sequence numbers
* Remove more instances of sequence_num
* Handle parsing sequnce_num from v5.4
Original commit: elastic/x-pack-elasticsearch@e60b206daf
This commit simplifies the WatchStatsTests to use a MockScriptPlugin.
The latch script engine previously depended on a static instance of the
engine to contain the latches. These are now moved to statics of the
test class itself.
Original commit: elastic/x-pack-elasticsearch@4170cd1bd3
This commit enables security to work with an index named .security (as
it could before) OR an alias named .security that points to a concrete
index by a different name that has the security index. This prepares
the ability to migrate from a 5.x to 6.x security index that allows
changing and re-indexing the underlying security index while maintaining
a .security alias that points to the underlying updated index.
relates elastic/x-pack-elasticsearch#1216
Original commit: elastic/x-pack-elasticsearch@9fee12e5a0
Suppress many job/datafeed errors if a node is known to be shutting down. Also, ensure started datafeeds and open jobs don't end up stopped/failed due to errors as the shutdown progresses, as this would prevent them automatically relocating to a different node.
relates elastic/x-pack-elasticsearch#1114
Original commit: elastic/x-pack-elasticsearch@e56a7dbea1
Set job to failed if autodetect manager fails closing, fix force closing of jobs that hang in closing
state, set timeout when waiting for clusterstate update, disallow closing of failed jobs with normal
close
relates elastic/x-pack-elasticsearch#1453
Original commit: elastic/x-pack-elasticsearch@493cf85e22
Aliases might be contained in requests that check index permissions
to disable caches etc. This commit preserves permissions for
aliases as well.
Original commit: elastic/x-pack-elasticsearch@233195aeba
The authentication object was changed in 5.4.0 in that it was conditionally signed depending on
the version and other factors. A bug was introduced however that causes the authentication to
actually get written with the version of the node it is being sent to even if that version is
greater than the version of the current node, which causes rolling upgrades to fail.
Original commit: elastic/x-pack-elasticsearch@a718ff8a52
* Add sort parameter for get buckets
* Add secondary sort by time
* Use default values from actions in rest requests
Original commit: elastic/x-pack-elasticsearch@a530c0bed6
This commit ensures the old 5.x index templates are removed
using the existing plugin hook, instead of the self written part.
Original commit: elastic/x-pack-elasticsearch@6faf08d98d
This commit adds compatibility checks while opening a job:
- Checks that jobs without versions (< 5.5) are not opened
- Checks that jobs with incompatible types are not opened
Original commit: elastic/x-pack-elasticsearch@a3adab733e
* Hide ML actions for tribe node client
* Remove unused parameters
* Enable ML actions and rest endpoints for the transport client
* Create the ML components for the transport client
* Add ml transport client tests
Original commit: elastic/x-pack-elasticsearch@509007ca29
* Remove repeated calls to validateAndReturnJobTask
* Wait for closing job
* Refactor resolving job ids
* More close job unit tests
* Don’t finalise closing jobs twice
Original commit: elastic/x-pack-elasticsearch@20616d6f0a
The LocalExporterTests.testLocalExporterFlush() test was removed in elastic/x-pack-elasticsearch#835 when
LocalExporterTests was changed. This test verified that export exceptions are
thrown when monitoring documents are exported using the Monitoring Bulk API but
the underlying monitoring indices are closed.
This commit reintroduces this test, but as a REST test this time.
relates elastic/x-pack-elasticsearch#416
Original commit: elastic/x-pack-elasticsearch@0a42f9a1be
This is the xpack side of elastic/elasticsearch#24447. The one caveat is
there are a number of places within the xpack api that use
BytesReference to pass down to templates. These are convert to encode
into a BytesArray when necessary, so as not to require changing all of
those apis here at once, but they should all be convert to String as
well.
Original commit: elastic/x-pack-elasticsearch@8399b9d8c3
For user tokens, we were storing the expiration time as a date with the date time format in the
mappings. Occasionally, we would get CI failures due to parsing the date and having an invalid
format. This change simplifies the user tokens to simply use an Instant and convert it to the epoch
millis when we index the document. This eliminates the need for a date formatter and should ensure
we no longer have issues with parsing the dates.
Additionally, the TokenAuthIntegTests#testExpireMultipleTimes could fail if the token was expired
but the approximation for the expiration time was after the current time. In order to resolve this
we get the exact expiration time from the token and use that in the test.
relates elastic/x-pack-elasticsearch#1255
Original commit: elastic/x-pack-elasticsearch@d4bf53e7bc
When we update a model snapshot we need to write it back to
the exact index where we read it from. This is necessary
for rollover as otherwise we would end up with two different
versions of the same snapshot.
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@b5d1ab38a7
Support the resolution of remote index names, including those that contain wildcards in the cluster name or index part)
Specifically these work:
- `GET /remote*:foo/_search`
- `GET /*:foo/_search`
- `GET /*:foo,*/_search`
- `GET /remote:*/_search`
- `GET /*:*/_search`
This change assumes that every user is allowed to attempt a cross-cluster search against any remote index, and the actual authorisation of indices happens on the remote nodes. Thus ` GET /*:foo/_search` will expand to search the `foo` index on every registered remote without consideration of the roles and privileges that the user has on the source cluster.
Original commit: elastic/x-pack-elasticsearch@b45041aaa3
As this does not require any reindexing this is easy to fix by just
changing the watch history template.
In addition the old templates are deleted on start up and the new ones
are instantiated.
Original commit: elastic/x-pack-elasticsearch@7e1ad495ad
I broke this by adding more randomization to core's randomTimeValue
method. Now it produces valid time values that don't round trip
properly over the XContent API. This change causes the test to skip
trying to round trip time these sorts of time values.
Original commit: elastic/x-pack-elasticsearch@dcdd588bdb
This is in preparation of introducing a write alias.
It adjusts all requests to persist results to do so
using a method that returns the write alias (even though
it currently returns the same as the read alias).
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@1358dd8dcf
This commit updates the RunAsIntegTests to randomly assign the superuser role to the user that
is authenticating with the cluster but the request is being run as a different user. This provides
additional validation that the authorization errors are actually coming from the user the request
is running as and not due to the authenticating user's privileges.
Original commit: elastic/x-pack-elasticsearch@c6360d13e6
Each job introduces new fields to the results index matching
the analysis terms. When the job is created, mappings for those
are added explicitly. However, when rollover is introduced, that
will not be the case. This commit prepares for that by adding
dynamic mapping of new fields as keyword.
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@8f6cd09a71
This was the behaviour in Shield 2.x, but it was accidentally changed during migration to X-Pack 5.x
Original commit: elastic/x-pack-elasticsearch@de0bf5e688
It just wastes 20 seconds while we timeout trying to open named pipes that cannot
possibly have been created.
Original commit: elastic/x-pack-elasticsearch@4e447874f6
fixes build errors. Still ensures that the timestamp is set to 'now' if the parsed logfile misses it.
Original commit: elastic/x-pack-elasticsearch@cf60e8d76b
If invalid job configs are transported to the master node then the root
cause of the validation exception gets reported as a remote_transport_exception,
which is extremely confusing.
This commit moves the validation of job configurations to the first node that
handles the action.
Fixeselastic/x-pack-kibana#1172
Original commit: elastic/x-pack-elasticsearch@5ed59d2a6f
Before searching for documents in monitoring indices, we need to ensure
that they exist and are available.
Original commit: elastic/x-pack-elasticsearch@29db55a1fe
This commit adds a base rest handler for security that handles the license checking in the security
apis. This was done previously in some rest handlers but not all and actually had issues where a
value would be returned but we may not have consumed all of the request parameters, which could
lead to a different response being returned than what we would have expected.
relates elastic/x-pack-elasticsearch#1236
Original commit: elastic/x-pack-elasticsearch@2f1100b64a
Adds defenseagainst broken scrolls to the fetching roles and users.
While Elasticsearch *shouldn't* send broken scrolls it has done so
in the past and when it does this causes security to consume the
entire heap and crash. This changes it so we instead fail the request
with a message about the scroll being broken.
Relates to elastic/x-pack-elasticsearch#1299
Original commit: elastic/x-pack-elasticsearch@dfef87e757
Some JDKs do not support the ECDSA cipher suites that we use in the EllipticCurveSSLTests, which
is the underlying cause of some CI failures. This change ensures there is at least one enabled
ECDSA cipher before testing that a connection can be made.
relates elastic/x-pack-elasticsearch#1278
Original commit: elastic/x-pack-elasticsearch@f6c93d776c
* [ML] Not an error to close a job twice
* Error if job is opening
* Address review comments
* Test closed job isn’t resolved
Original commit: elastic/x-pack-elasticsearch@7da7b24c08
Jason Tedor
Tim Vernum
Ali Beyad
Tim Brooks
Adrien Grand
Alexander Reelsen
markharwood
Simon Willnauer
David Roberts
Tim Sullivan
Andrew Cholakian
Jay Modi
Luca Cavanna
Igor Motov
jaymode
Yannick Welsch
Tim Vernum
Chris Earle
Colin Goodheart-Smithe
Brandon Kobel
Jack Conradson
Hendrik Muhs
Tal Levy
Nik Everett
Lisa Cawley
Dimitris Athanasiou
Lee Hinman
Christoph Büscher
Ryan Ernst
Suyog Rao
Daniel Mitterdorfer
David Kyle
Andy Bristol
Martijn van Groningen
Shaunak Kashyap
Pier-Hugues Pellerin
Jim Ferenczi
Boaz Leskes
Chris Earle
Tanguy Leroux
javanna