mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-15 09:25:40 +00:00
38886e8f23
This commit adds troubleshooting section for Kerberos. Most of the times the problems seen are caused due to invalid configurations like keytab missing principals or credentials not up to date. Time synchronization is an important part for Kerberos infrastructure and the time skew can cause problems. To debug further documentation explains how to enable JAAS Kerberos login module debugging and Kerberos/SPNEGO debugging by setting JVM system properties.
491 lines
18 KiB
Plaintext
491 lines
18 KiB
Plaintext
[role="xpack"]
|
|
[[security-troubleshooting]]
|
|
== {security} Troubleshooting
|
|
++++
|
|
<titleabbrev>{security}</titleabbrev>
|
|
++++
|
|
|
|
Use the information in this section to troubleshoot common problems and find
|
|
answers for frequently asked questions.
|
|
|
|
* <<security-trb-settings>>
|
|
* <<security-trb-roles>>
|
|
* <<security-trb-extraargs>>
|
|
* <<trouble-shoot-active-directory>>
|
|
* <<trb-security-maccurl>>
|
|
* <<trb-security-sslhandshake>>
|
|
* <<trb-security-ssl>>
|
|
* <<trb-security-kerberos>>
|
|
* <<trb-security-internalserver>>
|
|
* <<trb-security-setup>>
|
|
|
|
|
|
To get help, see <<help>>.
|
|
|
|
[[security-trb-settings]]
|
|
=== Some settings are not returned via the nodes settings API
|
|
|
|
*Symptoms:*
|
|
|
|
* When you use the {ref}/cluster-nodes-info.html[nodes info API] to retrieve
|
|
settings for a node, some information is missing.
|
|
|
|
*Resolution:*
|
|
|
|
This is intentional. Some of the settings are considered to be highly
|
|
sensitive: all `ssl` settings, ldap `bind_dn`, and `bind_password`.
|
|
For this reason, we filter these settings and do not expose them via
|
|
the nodes info API rest endpoint. You can also define additional
|
|
sensitive settings that should be hidden using the
|
|
`xpack.security.hide_settings` setting. For example, this snippet
|
|
hides the `url` settings of the `ldap1` realm and all settings of the
|
|
`ad1` realm.
|
|
|
|
[source, yaml]
|
|
------------------------------------------
|
|
xpack.security.hide_settings: xpack.security.authc.realms.ldap1.url,
|
|
xpack.security.authc.realms.ad1.*
|
|
------------------------------------------
|
|
|
|
[[security-trb-roles]]
|
|
=== Authorization exceptions
|
|
|
|
*Symptoms:*
|
|
|
|
* I configured the appropriate roles and the users, but I still get an
|
|
authorization exception.
|
|
* I can authenticate to LDAP, but I still get an authorization exception.
|
|
|
|
|
|
*Resolution:*
|
|
|
|
. Verify that the role names associated with the users match the roles defined
|
|
in the `roles.yml` file. You can use the `elasticsearch-users` tool to list all
|
|
the users. Any unknown roles are marked with `*`.
|
|
+
|
|
--
|
|
[source, shell]
|
|
------------------------------------------
|
|
bin/elasticsearch-users list
|
|
rdeniro : admin
|
|
alpacino : power_user
|
|
jacknich : monitoring,unknown_role* <1>
|
|
------------------------------------------
|
|
<1> `unknown_role` was not found in `roles.yml`
|
|
|
|
For more information about this command, see the
|
|
{ref}/users-command.html[`elasticsearch-users` command].
|
|
--
|
|
|
|
. If you are authenticating to LDAP, a number of configuration options can cause
|
|
this error.
|
|
+
|
|
--
|
|
|======================
|
|
|_group identification_ |
|
|
|
|
Groups are located by either an LDAP search or by the "memberOf" attribute on
|
|
the user. Also, If subtree search is turned off, it will search only one
|
|
level deep. See the <<ldap-settings, LDAP Settings>> for all the options.
|
|
There are many options here and sticking to the defaults will not work for all
|
|
scenarios.
|
|
|
|
| _group to role mapping_|
|
|
|
|
Either the `role_mapping.yml` file or the location for this file could be
|
|
misconfigured. See <<security-files, Security Files>> for more.
|
|
|
|
|_role definition_|
|
|
|
|
The role definition might be missing or invalid.
|
|
|
|
|======================
|
|
|
|
To help track down these possibilities, add the following lines to the end of
|
|
the `log4j2.properties` configuration file in the `ES_PATH_CONF`:
|
|
|
|
[source,properties]
|
|
----------------
|
|
logger.authc.name = org.elasticsearch.xpack.security.authc
|
|
logger.authc.level = DEBUG
|
|
----------------
|
|
|
|
A successful authentication should produce debug statements that list groups and
|
|
role mappings.
|
|
--
|
|
|
|
[[security-trb-extraargs]]
|
|
=== Users command fails due to extra arguments
|
|
|
|
*Symptoms:*
|
|
|
|
* The `elasticsearch-users` command fails with the following message:
|
|
`ERROR: extra arguments [...] were provided`.
|
|
|
|
*Resolution:*
|
|
|
|
This error occurs when the `elasticsearch-users` tool is parsing the input and
|
|
finds unexpected arguments. This can happen when there are special characters
|
|
used in some of the arguments. For example, on Windows systems the `,` character
|
|
is considered a parameter separator; in other words `-r role1,role2` is
|
|
translated to `-r role1 role2` and the `elasticsearch-users` tool only
|
|
recognizes `role1` as an expected parameter. The solution here is to quote the
|
|
parameter: `-r "role1,role2"`.
|
|
|
|
For more information about this command, see
|
|
{ref}/users-command.html[`elasticsearch-users` command].
|
|
|
|
[[trouble-shoot-active-directory]]
|
|
=== Users are frequently locked out of Active Directory
|
|
|
|
*Symptoms:*
|
|
|
|
* Certain users are being frequently locked out of Active Directory.
|
|
|
|
*Resolution:*
|
|
|
|
Check your realm configuration; realms are checked serially, one after another.
|
|
If your Active Directory realm is being checked before other realms and there
|
|
are usernames that appear in both Active Directory and another realm, a valid
|
|
login for one realm might be causing failed login attempts in another realm.
|
|
|
|
For example, if `UserA` exists in both Active Directory and a file realm, and
|
|
the Active Directory realm is checked first and file is checked second, an
|
|
attempt to authenticate as `UserA` in the file realm would first attempt to
|
|
authenticate against Active Directory and fail, before successfully
|
|
authenticating against the `file` realm. Because authentication is verified on
|
|
each request, the Active Directory realm would be checked - and fail - on each
|
|
request for `UserA` in the `file` realm. In this case, while the authentication
|
|
request completed successfully, the account on Active Directory would have
|
|
received several failed login attempts, and that account might become
|
|
temporarily locked out. Plan the order of your realms accordingly.
|
|
|
|
Also note that it is not typically necessary to define multiple Active Directory
|
|
realms to handle domain controller failures. When using Microsoft DNS, the DNS
|
|
entry for the domain should always point to an available domain controller.
|
|
|
|
|
|
[[trb-security-maccurl]]
|
|
=== Certificate verification fails for curl on Mac
|
|
|
|
*Symptoms:*
|
|
|
|
* `curl` on the Mac returns a certificate verification error even when the
|
|
`--cacert` option is used.
|
|
|
|
|
|
*Resolution:*
|
|
|
|
Apple's integration of `curl` with their keychain technology disables the
|
|
`--cacert` option.
|
|
See http://curl.haxx.se/mail/archive-2013-10/0036.html for more information.
|
|
|
|
You can use another tool, such as `wget`, to test certificates. Alternately, you
|
|
can add the certificate for the signing certificate authority MacOS system
|
|
keychain, using a procedure similar to the one detailed at the
|
|
http://support.apple.com/kb/PH14003[Apple knowledge base]. Be sure to add the
|
|
signing CA's certificate and not the server's certificate.
|
|
|
|
|
|
[[trb-security-sslhandshake]]
|
|
=== SSLHandshakeException causes connections to fail
|
|
|
|
*Symptoms:*
|
|
|
|
* A `SSLHandshakeException` causes a connection to a node to fail and indicates
|
|
that there is a configuration issue. Some of the common exceptions are shown
|
|
below with tips on how to resolve these issues.
|
|
|
|
|
|
*Resolution:*
|
|
|
|
`java.security.cert.CertificateException: No name matching node01.example.com found`::
|
|
+
|
|
--
|
|
Indicates that a client connection was made to `node01.example.com` but the
|
|
certificate returned did not contain the name `node01.example.com`. In most
|
|
cases, the issue can be resolved by ensuring the name is specified during
|
|
certificate creation. For more information, see <<ssl-tls>>. Another scenario is
|
|
when the environment does not wish to use DNS names in certificates at all. In
|
|
this scenario, all settings in `elasticsearch.yml` should only use IP addresses
|
|
including the `network.publish_host` setting.
|
|
--
|
|
|
|
`java.security.cert.CertificateException: No subject alternative names present`::
|
|
+
|
|
--
|
|
Indicates that a client connection was made to an IP address but the returned
|
|
certificate did not contain any `SubjectAlternativeName` entries. IP addresses
|
|
are only used for hostname verification if they are specified as a
|
|
`SubjectAlternativeName` during certificate creation. If the intent was to use
|
|
IP addresses for hostname verification, then the certificate will need to be
|
|
regenerated with the appropriate IP address. See <<ssl-tls>>.
|
|
--
|
|
|
|
`javax.net.ssl.SSLHandshakeException: null cert chain` and `javax.net.ssl.SSLException: Received fatal alert: bad_certificate`::
|
|
+
|
|
--
|
|
The `SSLHandshakeException` indicates that a self-signed certificate was
|
|
returned by the client that is not trusted as it cannot be found in the
|
|
`truststore` or `keystore`. This `SSLException` is seen on the client side of
|
|
the connection.
|
|
--
|
|
|
|
`sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` and `javax.net.ssl.SSLException: Received fatal alert: certificate_unknown`::
|
|
+
|
|
--
|
|
This `SunCertPathBuilderException` indicates that a certificate was returned
|
|
during the handshake that is not trusted. This message is seen on the client
|
|
side of the connection. The `SSLException` is seen on the server side of the
|
|
connection. The CA certificate that signed the returned certificate was not
|
|
found in the `keystore` or `truststore` and needs to be added to trust this
|
|
certificate.
|
|
--
|
|
|
|
[[trb-security-ssl]]
|
|
=== Common SSL/TLS exceptions
|
|
|
|
*Symptoms:*
|
|
|
|
* You might see some exceptions related to SSL/TLS in your logs. Some of the
|
|
common exceptions are shown below with tips on how to resolve these issues. +
|
|
|
|
|
|
|
|
*Resolution:*
|
|
|
|
`WARN: received plaintext http traffic on a https channel, closing connection`::
|
|
+
|
|
--
|
|
Indicates that there was an incoming plaintext http request. This typically
|
|
occurs when an external applications attempts to make an unencrypted call to the
|
|
REST interface. Please ensure that all applications are using `https` when
|
|
calling the REST interface with SSL enabled.
|
|
--
|
|
|
|
`org.elasticsearch.common.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:`::
|
|
+
|
|
--
|
|
Indicates that there was incoming plaintext traffic on an SSL connection. This
|
|
typically occurs when a node is not configured to use encrypted communication
|
|
and tries to connect to nodes that are using encrypted communication. Please
|
|
verify that all nodes are using the same setting for
|
|
`xpack.security.transport.ssl.enabled`.
|
|
|
|
For more information about this setting, see
|
|
{ref}/security-settings.html[Security Settings in {es}].
|
|
--
|
|
|
|
`java.io.StreamCorruptedException: invalid internal transport message format, got`::
|
|
+
|
|
--
|
|
Indicates an issue with data received on the transport interface in an unknown
|
|
format. This can happen when a node with encrypted communication enabled
|
|
connects to a node that has encrypted communication disabled. Please verify that
|
|
all nodes are using the same setting for `xpack.security.transport.ssl.enabled`.
|
|
|
|
For more information about this setting, see
|
|
{ref}/security-settings.html[Security Settings in {es}].
|
|
--
|
|
|
|
`java.lang.IllegalArgumentException: empty text`::
|
|
+
|
|
--
|
|
This exception is typically seen when a `https` request is made to a node that
|
|
is not using `https`. If `https` is desired, please ensure the following setting
|
|
is in `elasticsearch.yml`:
|
|
|
|
[source,yaml]
|
|
----------------
|
|
xpack.security.http.ssl.enabled: true
|
|
----------------
|
|
|
|
For more information about this setting, see
|
|
{ref}/security-settings.html[Security Settings in {es}].
|
|
--
|
|
|
|
`ERROR: unsupported ciphers [...] were requested but cannot be used in this JVM`::
|
|
+
|
|
--
|
|
This error occurs when a SSL/TLS cipher suite is specified that cannot supported
|
|
by the JVM that {es} is running in. Security tries to use the specified cipher
|
|
suites that are supported by this JVM. This error can occur when using the
|
|
Security defaults as some distributions of OpenJDK do not enable the PKCS11
|
|
provider by default. In this case, we recommend consulting your JVM
|
|
documentation for details on how to enable the PKCS11 provider.
|
|
|
|
Another common source of this error is requesting cipher suites that use
|
|
encrypting with a key length greater than 128 bits when running on an Oracle JDK.
|
|
In this case, you must install the
|
|
<<ciphers, JCE Unlimited Strength Jurisdiction Policy Files>>.
|
|
--
|
|
|
|
[[trb-security-kerberos]]
|
|
=== Common Kerberos exceptions
|
|
|
|
*Symptoms:*
|
|
|
|
* User authentication fails due to either GSS negotiation failure
|
|
or a service login failure (either on the server or in the {es} http client).
|
|
Some of the common exceptions are listed below with some tips to help resolve
|
|
them.
|
|
|
|
*Resolution:*
|
|
|
|
`Failure unspecified at GSS-API level (Mechanism level: Checksum failed)`::
|
|
+
|
|
--
|
|
|
|
When you see this error message on the HTTP client side, then it may be
|
|
related to an incorrect password.
|
|
|
|
When you see this error message in the {es} server logs, then it may be
|
|
related to the {es} service keytab. The keytab file is present but it failed
|
|
to log in as the user. Please check the keytab expiry. Also check whether the
|
|
keytab contain up-to-date credentials; if not, replace them.
|
|
|
|
You can use tools like `klist` or `ktab` to list principals inside
|
|
the keytab and validate them. You can use `kinit` to see if you can acquire
|
|
initial tickets using the keytab. Please check the tools and their documentation
|
|
in your Kerberos environment.
|
|
|
|
Kerberos depends on proper hostname resolution, so please check your DNS infrastructure.
|
|
Incorrect DNS setup, DNS SRV records or configuration for KDC servers in `krb5.conf`
|
|
can cause problems with hostname resolution.
|
|
|
|
--
|
|
|
|
`Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))`::
|
|
|
|
`Failure unspecified at GSS-API level (Mechanism level: Clock skew too great (37))`::
|
|
+
|
|
--
|
|
|
|
To prevent replay attacks, Kerberos V5 sets a maximum tolerance for computer
|
|
clock synchronization and it is typically 5 minutes. Please check whether
|
|
the time on the machines within the domain is in sync.
|
|
|
|
--
|
|
|
|
As Kerberos logs are often cryptic in nature and many things can go wrong
|
|
as it depends on external services like DNS and NTP. You might
|
|
have to enable additional debug logs to determine the root cause of the issue.
|
|
|
|
{es} uses a JAAS (Java Authentication and Authorization Service) Kerberos login
|
|
module to provide Kerberos support. To enable debug logs on {es} for the login
|
|
module use following Kerberos realm setting:
|
|
[source,yaml]
|
|
----------------
|
|
xpack.security.authc.realms.<realm-name>.krb.debug: true
|
|
----------------
|
|
|
|
For detailed information, see {ref}/security-settings.html#ref-kerberos-settings[Kerberos realm settings].
|
|
|
|
Sometimes you may need to go deeper to understand the problem during SPNEGO
|
|
GSS context negotiation or look at the Kerberos message exchange. To enable
|
|
Kerberos/SPNEGO debug logging on JVM, add following JVM system properties:
|
|
|
|
`-Dsun.security.krb5.debug=true`
|
|
|
|
`-Dsun.security.spnego.debug=true`
|
|
|
|
For more information about JVM system properties, see {ref}/jvm-options.html[configuring JVM options].
|
|
|
|
[[trb-security-internalserver]]
|
|
=== Internal Server Error in Kibana
|
|
|
|
*Symptoms:*
|
|
|
|
* In 5.1.1, an `UnhandledPromiseRejectionWarning` occurs and {kib} displays an
|
|
Internal Server Error.
|
|
//TBD: Is the same true for later releases?
|
|
|
|
*Resolution:*
|
|
|
|
If the Security plugin is enabled in {es} but disabled in {kib}, you must
|
|
still set `elasticsearch.username` and `elasticsearch.password` in `kibana.yml`.
|
|
Otherwise, {kib} cannot connect to {es}.
|
|
|
|
|
|
[[trb-security-setup]]
|
|
=== Setup-passwords command fails due to connection failure
|
|
|
|
The {ref}/setup-passwords.html[elasticsearch-setup-passwords command] sets
|
|
passwords for the built-in users by sending user management API requests. If
|
|
your cluster uses SSL/TLS for the HTTP (REST) interface, the command attempts to
|
|
establish a connection with the HTTPS protocol. If the connection attempt fails,
|
|
the command fails.
|
|
|
|
*Symptoms:*
|
|
|
|
. {es} is running HTTPS, but the command fails to detect it and returns the
|
|
following errors:
|
|
+
|
|
--
|
|
[source, shell]
|
|
------------------------------------------
|
|
Cannot connect to elasticsearch node.
|
|
java.net.SocketException: Unexpected end of file from server
|
|
...
|
|
ERROR: Failed to connect to elasticsearch at
|
|
http://127.0.0.1:9200/_xpack/security/_authenticate?pretty.
|
|
Is the URL correct and elasticsearch running?
|
|
------------------------------------------
|
|
--
|
|
|
|
. SSL/TLS is configured, but trust cannot be established. The command returns
|
|
the following errors:
|
|
+
|
|
--
|
|
[source, shell]
|
|
------------------------------------------
|
|
SSL connection to
|
|
https://127.0.0.1:9200/_xpack/security/_authenticate?pretty
|
|
failed: sun.security.validator.ValidatorException:
|
|
PKIX path building failed:
|
|
sun.security.provider.certpath.SunCertPathBuilderException:
|
|
unable to find valid certification path to requested target
|
|
Please check the elasticsearch SSL settings under
|
|
xpack.security.http.ssl.
|
|
...
|
|
ERROR: Failed to establish SSL connection to elasticsearch at
|
|
https://127.0.0.1:9200/_xpack/security/_authenticate?pretty.
|
|
------------------------------------------
|
|
--
|
|
|
|
. The command fails because hostname verification fails, which results in the
|
|
following errors:
|
|
+
|
|
--
|
|
[source, shell]
|
|
------------------------------------------
|
|
SSL connection to
|
|
https://idp.localhost.test:9200/_xpack/security/_authenticate?pretty
|
|
failed: java.security.cert.CertificateException:
|
|
No subject alternative DNS name matching
|
|
elasticsearch.example.com found.
|
|
Please check the elasticsearch SSL settings under
|
|
xpack.security.http.ssl.
|
|
...
|
|
ERROR: Failed to establish SSL connection to elasticsearch at
|
|
https://elasticsearch.example.com:9200/_xpack/security/_authenticate?pretty.
|
|
------------------------------------------
|
|
--
|
|
|
|
*Resolution:*
|
|
|
|
. If your cluster uses TLS/SSL for the HTTP interface but the
|
|
`elasticsearch-setup-passwords` command attempts to establish a non-secure
|
|
connection, use the `--url` command option to explicitly specify an HTTPS URL.
|
|
Alternatively, set the `xpack.security.http.ssl.enabled` setting to `true`.
|
|
|
|
. If the command does not trust the {es} server, verify that you configured the
|
|
`xpack.security.http.ssl.certificate_authorities` setting or the
|
|
`xpack.security.http.ssl.truststore.path` setting.
|
|
|
|
. If hostname verification fails, you can disable this verification by setting
|
|
`xpack.security.http.ssl.verification_mode` to `certificate`.
|
|
|
|
For more information about these settings, see
|
|
{ref}/security-settings.html[Security Settings in {es}].
|