Since change elastic/x-pack-elasticsearch@f796949 authentication is not allowed to respond with null, it must be AuthenticationResult.notHandled()
- Fixes 1 case where the native realm would respond null if the user was not found
- Fixes some edge cases in the LDAP realm.
Original commit: elastic/x-pack-elasticsearch@bc739a1d40
In logstash parlance there really is no pipeline.name, its pipeline.id
This also removes support for deprecated gauge types `text` and `boolean` we will be removing those soon in logstash
This also renames `counters` to `long_counters` to be more explicit and for consistency with the gauge type. Also, if we ever decide to add other types of counters this will be more clear
Original commit: elastic/x-pack-elasticsearch@8f44a94579
The user/role was added to simplify they setup surrounding Beats monitoring, but Beats monitoring has been delayed until the UI work is begun.
Original commit: elastic/x-pack-elasticsearch@1c0c85562e
When security initialization fails during startup today we throw an
Error. This triggers the uncaught exception handler immediately killing
the node. While the node is going to die either way, we should not be
triggering the killer and in general we should avoid using Throwables
that extend Error.
Relates elastic/x-pack-elasticsearch#2035
Original commit: elastic/x-pack-elasticsearch@546f7f9002
This commit removes a legacy check for an unsupported environment
variable. This environment variable has not been supported since 1.x so
it is safe to stop checking for the existence of this setting.
Relates elastic/x-pack-elasticsearch#2048
Original commit: elastic/x-pack-elasticsearch@023230fa9e
Today we explicitly export the HOSTNAME variable from scripts. This is
probably a relic from the days when the scripts were not run on bash but
instead assume a POSIX-compliant shell only where HOSTNAME is not
guaranteed to exist. Yet, bash guarantees that HOSTNAME is set so we do
not need to set it in scripts. This commit removes this legacy.
Relates elastic/x-pack-elasticsearch#2047
Original commit: elastic/x-pack-elasticsearch@7b833e061c
This commit fixes the croneval script to respect the CONF_DIR
environment variable used to locate the configuration directory.
Original commit: elastic/x-pack-elasticsearch@79974947f9
This commit fixes an issue with the usage of the environment variable
ES_HOME in the certgen script; the script was missing the use of $ to
obtain the value of the environment variable ES_HOME.
Relates elastic/x-pack-elasticsearch#2046
Original commit: elastic/x-pack-elasticsearch@63128db0eb
With the new template and mapping update mechanisms introduced as part
of the Upgrade API work, the NativeRealmMigrator is no longer needed or
used. This commit removes the NativeRealmMigrator code and the
associated tests for it.
Original commit: elastic/x-pack-elasticsearch@5d2d7a582c
This is related to elastic/x-pack-elasticsearch#1778. This commit adds a setting
(xpack.license.self_generated.type) which allows the user to specify
what type of license will be self-generated on node startup. The allowed
types are basic or trial.
Original commit: elastic/x-pack-elasticsearch@0a16a59e10
To achieve backwards compatibility the easiest way is
to restore the old watcher stats, which are supposed to run
on the master node only.
The distributed watcher stats have been moved under the statsdist
package and the action name has been changed as well.
This way there is no need to have a serialization BWC layer,
we can just call different actions.
Note: With the current approach developers still need to change
their java applications if they try to receive watcher stats,
as by default we are now using the distributed stats in the
watcher client.
Original commit: elastic/x-pack-elasticsearch@49b3a45452
Closing jobs can still use some or all of the threads that communicate
with the C++ process - the number of threads used will decrease as the
close progresses, but at the beginning of the closure all are still in
use. Therefore, to prevent the risk of EsRejectedExecution exceptions
for the autodetect communications threadpool, closing jobs need to be
considered when checking that enough threads exist to start a new
process. An explicit check produces a much more understandable error
message than an EsRejectedExecution exception.
relates elastic/x-pack-elasticsearch#1364
Original commit: elastic/x-pack-elasticsearch@845bfe0188
This is important for two reasons:
1. If a job hangs in the closing state for any reason there is now a
way to get rid of it
2. The force delete endpoint (as used by the UI) killed open jobs before
deleting them, but could not kill closing jobs, which created the
possibility that if a job was deleted from the UI while in the closing
state then the last few results could be indexed after the deletion
completed
relates elastic/x-pack-elasticsearch#1796
Original commit: elastic/x-pack-elasticsearch@1471106e06
Add an option to the ssl certificate generation tool (certgen) that generates PKCSelastic/x-pack-elasticsearch#12 (.p12) files in addition to the certificate (.crt) and key (.key) files.
A PKCSelastic/x-pack-elasticsearch#12 store is a container format for storing multiple crypto objects in a single file, which means we can put the cert and key into the same file.
These format is particularly useful for .NET environments, where .NET Core requires a single into file for PKI authentication.
Also adds documentation for all the command-line options in certgen.
Original commit: elastic/x-pack-elasticsearch@d10f88f12d
* [Monitoring] Add Cluster Alert for X-Pack License Expiration
* work on scripts round 1
* updates per feedback
* spaces
* fix NPE error in transform
* condition to allow updating metadata in the alerts index in every interval
* custom subject message
* update name of indexing actions
* ensure ctx.metadata is updated even if alert is not resolved
* fix omission of absoluteTime
* skip info-level alerts for trial-type license
* move break above `fromNow` declaration
* fix test
Original commit: elastic/x-pack-elasticsearch@f13718f5b5
the upgrade API is lacking some functionality in a special case,
where triggered_watches exists, but .watches does not. This
deletes the triggered watches index manually until we integrated
this properly in the upgrade API to fix the tests
Original commit: elastic/x-pack-elasticsearch@e9d1b0d35d
This commit adds the upgrade API functionality and script for security.
It also enables previously muted tests that would fail due to the lack
of security upgrade features in testing cluster restarts and old
security index backward compatibility.
Original commit: elastic/x-pack-elasticsearch@4abe9f1263
This change makes the restricted trust configuration delegate the list of files to monitor to the
trust configuration that it wraps so that all files that should be monitored for changes are
monitored for changes.
Relates elastic/x-pack-elasticsearch#1919
Original commit: elastic/x-pack-elasticsearch@227db92ac0
We were catching IndexNotFoundException, which was hiding the fact that delete index and update aliases APIs don't accept aliases anymore. Now that the exception changed this problem popped up. We now rather call get index providing .security as index name, then delete the concrete indices returned in the response.
Original commit: elastic/x-pack-elasticsearch@18f64f9a41
This change fixes some cases where calls to the LDAP library can result in a new connection being
created that were not wrapped in privileged connect calls. This would result in permission denied
errors when trying to make the connection.
Original commit: elastic/x-pack-elasticsearch@182c790dd4
This changes the SecurityTribeIT tests to use the same settings object when creating the settings
for the tribe node. Previously two different objects were being created and we would read regular
settings from one and secure settings from another. This causes problems since randomization means
that there may be settings added on the first call that do not get added on the second call. One
example is the randomization of when to only use a keystore or to use both a keystore and a
truststore. On the first call, we would add settings for both a keystore and a truststore but on
the second call only the keystore settings were added. This lead to failures as we would not be
able to open a password protected truststore since the password was never added to the secure
settings.
relates elastic/x-pack-elasticsearch#2005
Original commit: elastic/x-pack-elasticsearch@bbdb3ec662
This change clears the caches in the native realm and the composite roles store when there is a
a change in the health of the security index that necessitates this. When the security index goes
to a red state, the caches are left in tact as this allows for management operations to be
performed for a limited amount of time. When the index transitions out of the red state or exists
when it didn't exist before, the caches will be cleared so that we remove any stale values.
relates elastic/x-pack-elasticsearch#1789
Original commit: elastic/x-pack-elasticsearch@914959ea6b
This adds a bootstrap-check that makes it an error to configure a role mapping file that doesn't exist or cannot be parsed.
We are still lenient on dynamic reload because
(a) killing a running node is quite drastic
(b) file writes aren't atomic, so we might be picking up a file that is half way through being written (etc).
If you rely on the default role mapping filename, then it doesn't need to exist (because you might be using the role mapping API instead) but if it does exist it has to parse successfully
Original commit: elastic/x-pack-elasticsearch@5424dea4c4
The default for group_search.attribute is to search by DN, but explicitly setting that to dn
wouldn't work because the DN is returned in a special value in the result, and not in the attributes list.
This change detects when user_attribute is set to dn and treats it the same way as the default value.
Original commit: elastic/x-pack-elasticsearch@1933410a0b
- Changes the reloading test to always trust the "trusted" cert so that the health-ping works
- Adds some more logging in case we get new failures
Original commit: elastic/x-pack-elasticsearch@993bf9c721
This is related to elastic/x-pack-elasticsearch#1217. This commit requires that the elastic password
be bootstrapped for the user to be authenticated. As a result it removes
the special "setup" mode that allowed the user to be authenticated from
localhost.
Additionally, this commit updates the tests to work with this
functionality.
Original commit: elastic/x-pack-elasticsearch@d0d5d697a7
When a user asks for upgrade information for all indices and there are no indices in the cluster, upgrade assistance should just return an empty response indicating that no indices require upgrade or reindexing. This commit also reverts the temporary fix in WatchBackwardsCompatibilityIT tests that was added as a workaround for this issue.
Original commit: elastic/x-pack-elasticsearch@2ea9707867
Ensures that parent task is propagated to child operations to ensure that reindex operation can be cancelled if needed.
Original commit: elastic/x-pack-elasticsearch@fa40b5a951
This change fixes the creation of the bind DN string for active directory realms so that they do
not add the `@domain` suffix to the bind DN, when it is a actual DN value.
Original commit: elastic/x-pack-elasticsearch@bd04c07e16
This adds the Cloud detection mappings so that phone home can take advantage of their existence.
It also sets the system load fields to use `half_floats`.
Original commit: elastic/x-pack-elasticsearch@75f7992d38
This stops checking for older alerts now that we support emailing based on state changes. This only applies to 6.0 because the upgrade _can_ still happen usefully in 5.6 without the noise.
Original commit: elastic/x-pack-elasticsearch@9d73c64daa
The active directory user principal name format typically takes the form user@domain, which is what
the current implementation expects. However, active directory also allows the definition of other
suffixes that are not actual domains. A user can still authenticate using this user principal name
but the behavior of our realm would cause it to fail as it parsed the suffix as a domain and used it
as the search base for the user. Instead, we should use the default user search base and only look
for entries that have this exact user principal name. In a scenario where a realm is configured for
multiple domains in the same forest, the search base should be the base for the entire forest.
relates elastic/x-pack-elasticsearch#1744
Original commit: elastic/x-pack-elasticsearch@de00c4817e
Optional restrictions can be applied on top of an existing certificate trust scheme (PEM CAs, JKS TrustStore etc).
The restrictions reduce the set of certificate that would be otherwise trusted.
In this commit, the only supported restrictions are to filter by the certificate's SubjectAlternativeNames that are tagged as "other name" with an object-identifier of "cn"
Original commit: elastic/x-pack-elasticsearch@c6105a47df
This allows for messages to be returned, and distinguishes between 4 different results:
- I have authenticated the user
- I don't know how to authenticate that user. Try another realm.
- I tried to authenticate the user, but failed. Try another realm.
- I tried to authenticate the user, but failed. Fail the authentication attempt.
Original commit: elastic/x-pack-elasticsearch@f796949cfb
This commit adds support for a bind user when using the active directory realm. The addition of a
bind user also enables support for the user lookup mechanism, which is necessary to support the run
as functionality that we provide.
relates elastic/x-pack-elasticsearch#179
Original commit: elastic/x-pack-elasticsearch@40b07b3422
When there are data or master nodes in the cluster, that are older
than ES 6.0 alpha3, then watcher will only start on the master node.
Changed all transport actions to be master node actions, as there is
already a method to decide to run locally, which we can piggyback on.
Original commit: elastic/x-pack-elasticsearch@65cecb6d69
* [Monitoring] Update template version to 6000003
* update LAST_UPDATED_VERSION
* use the 6.0.0-beta1 version constant
* use the 6.0.0-beta1 version value (6000026)
* fix cluster uuid reference in terms filters
Original commit: elastic/x-pack-elasticsearch@40c822d396
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.
Original commit: elastic/x-pack-elasticsearch@926a7b2d86
In case of mixed version in a cluster the update could potentially downgrade the mappings. This CL changes the check to avoid this situation.
Original commit: elastic/x-pack-elasticsearch@9ecd5df70e
This commit provides an explicit name for the log tail thread, otherwise
the thread shows up with a generic name like Thread-12 in stack
dumps. While the stack trace provides enough information to conclude
what this thread is doing, it is better to be more explicit about the
purpose of this thread so that that can be discerned directly from the
name of the thread.
Additionally, we mark this thread as a daemon thread. Since this thread
is created by the main thread during node initialization it defaults to
being a user thread. Since the JVM only exits when the only threads
running are not user threads, if this thread were to somehow block then
it could prevent the server JVM from exiting. As such, this thread
should be marked as a daemon thread.
Relates elastic/x-pack-elasticsearch#1969
Original commit: elastic/x-pack-elasticsearch@b3761c7d91
This commit adds a setting to allow changing the user search filter. Previously the filter was a
simple equality filter that mapped a given attribute to the value of the username. The default
behavior remains the same with this change but provides additional flexibility to users to who may
need more advanced LDAP searches. The user attribute setting has been deprecated due to the overlap
with the new filter setting.
relates elastic/x-pack-elasticsearch#1861
Original commit: elastic/x-pack-elasticsearch@e9d797e81c
When having a mixed cluster with 5.6 and 6.0 nodes, the template upgrade
service has a cluster state listener that deletes the old watches and
triggered_watches index template. However during that time the 5.6 node
WatcherIndexTemplateRegistry checks if the templates are missing and
adds them back. This results in a race, because the new .watches index
template does not get added by the WatcherIndexTemplateRegistry when the
6.0 node is node a master node.
This commit circumvents this issue, by only deleting the watches and
triggered watches template during the upgrade process.
Original commit: elastic/x-pack-elasticsearch@71380f460a
Following elastic/elasticsearch#23997 this was only working due to
the way we were suppressing certain errors during job deletion.
This PR makes the situations we want to ignore during job deletion
clearer and adheres to the intention of elastic/elasticsearch#23997
by only specifying concrete indices to the `indices` arguments of
deletion calls.
Original commit: elastic/x-pack-elasticsearch@2458c3db40
If the cluster is still making cluster state updates while disabling the license, cluster state updates might not go through,
triggering an assertion failure at the end of the test that checks if all cluster states have been applied.
relates elastic/x-pack-elasticsearch#1627
Original commit: elastic/x-pack-elasticsearch@e11863fd02
Adds a update procedure for ML index mappings in order to allow adding new fields. The ES version is stored in the "_meta" field of the ML mappings which then get applied to every index. When opening a job, this version is checked on the (shared) index and a mapping update is performed in case that the version is older than current.
Original commit: elastic/x-pack-elasticsearch@211608c7ad
* add index checks
The following checks apply to both 5.x as well as 6.0
index checks:
- coercion of boolean fields
- the _all meta field is now disabled by default
- the include_in_all mapping parameter is now disallowed
- unrecognized match_mapping_type options not silently ignored
- similarity settings
- store settings
- store throttling settings
- shadow replicas have been removed
Original commit: elastic/x-pack-elasticsearch@5583b21763
This is related to elastic/x-pack-elasticsearch#1217. This commit adds a ClusterStateListener at
node startup. Once the cluster and security index are ready, this
listener will attempt to set the elastic user's password with the
bootstrap password pulled from the keystore. If the password is not in
the keystore or the elastic password has already been set, nothing will
be done.
Original commit: elastic/x-pack-elasticsearch@7fc4943c45
As the TemplateUpgradeService requires permissions to add and
delete index template, we have to grant those to the _system user.
This commit adds such permissions plus an integration test.
Original commit: elastic/x-pack-elasticsearch@a76ca9c738
The monitoring watches are roughly executing the same queries even when
they run against different clusters. However the way they were created,
where the cluster name is replaced via search & replace instead of using
watch metadata implies, that every watch is different from a script
compilation cache perspective. On top of that every of those watches is
executed once a minute. So if a new node becomes master and you monitor
three clusters, this results in a fair share of compilations in the first minute. The
reason for the compilation is the fact, that the search input uses
mustache for being able to add dynamic parts into a search using
mustache.
Several of those watches also need to compile more than one search
request.
The maximum default value for script compilations is only 15 and thus at
least one watch will not be executed due to failing script compilations.
This commit changes the four watches, so that the search requests are
cacheable. This means, no matter how many clusters you monitor, there
will be only needed four compilations for the different watches and
that's it.
Relates elastic/support-dev-help#2090
Original commit: elastic/x-pack-elasticsearch@6c877421bb
This test creates watches in old versions of elasticsearch, upgrades them after upgrading cluster to the latest version and then tests that they were upgraded correctly.
Original commit: elastic/x-pack-elasticsearch@b9d45eb2a5
* [Monitoring] Email actions for Cluster Alerts
* fix quotations in email fields
* move email vars to transform, and rename for snake_case
* add state to email subject for cluster status alert
* remove types field in kibana_settings search
* simplify email action condition script
* uppercase the state for the email subject
* only append state to email subject if alert is new
* show state in email subject even when alert is resolved
Original commit: elastic/x-pack-elasticsearch@e6fdd8d620
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.
This commit loosens two aspects of the security.
1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.
Original commit: elastic/x-pack-elasticsearch@8a2a577038
This is step 2 of elastic/x-pack-elasticsearch#1604
This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.
Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
ML has two types of custom cluster state:
1. jobs
2. datafeeds
These need to be parsed from JSON in two situations:
1. Create/update of the job/datafeed
2. Restoring cluster state on startup
Previously we used exactly the same parser in both situations, but this
severely limits our ability to add new features. This is because the
parser was very strict. This was good when accepting create/update
requests from users, but when restoring cluster state from disk it meant
that we could not add new fields, as that would prevent reloading in
mixed version clusters.
This commit introduces a second parser that tolerates unknown fields for
each object that is stored in cluster state. Then we use this more
tolerant parser when parsing cluster state, but still use the strict
parser when parsing REST requests.
relates elastic/x-pack-elasticsearch#1732
Original commit: elastic/x-pack-elasticsearch@754e51d1ec
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/24437
It changes two things, for the disable tests, it uses a valid endpoint instead
of a previously invalid endpoint that happened to return a 400 because the
endpoint was bad, regardless of if watcher was disabled.
The other change is to create the watches index by putting a watch using the
correct API, rather than manually creating the index. This is because
`RestHijackOperationAction` hijacks operations like this and stops accessing the
endpoint in a regular manner.
Original commit: elastic/x-pack-elasticsearch@3be78d9aea
Tim Vernum
Tim Brooks
Jason Tedor
Andrew Cholakian
Chris Earle
Jack Conradson
Simon Willnauer
Alexander Reelsen
Ali Beyad
Adrien Grand
markharwood
David Roberts
Tim Sullivan
Jay Modi
Luca Cavanna
Igor Motov
jaymode
Yannick Welsch
Tim Vernum
Colin Goodheart-Smithe
Brandon Kobel
Hendrik Muhs
Tal Levy
Nik Everett
Lisa Cawley
Dimitris Athanasiou
Lee Hinman