414 lines
15 KiB
Plaintext
414 lines
15 KiB
Plaintext
[[security-troubleshooting]]
|
|
== {security} Troubleshooting
|
|
|
|
Use the information in this section to troubleshoot common problems and find
|
|
answers for frequently asked questions.
|
|
|
|
* <<security-trb-settings>>
|
|
* <<security-trb-roles>>
|
|
* <<security-trb-extraargs>>
|
|
* <<trouble-shoot-active-directory>>
|
|
* <<trb-security-maccurl>>
|
|
* <<trb-security-sslhandshake>>
|
|
* <<trb-security-ssl>>
|
|
* <<trb-security-internalserver>>
|
|
* <<trb-security-setup>>
|
|
|
|
|
|
To get help, see <<xpack-help>>.
|
|
|
|
[[security-trb-settings]]
|
|
=== Some settings are not returned via the nodes settings API
|
|
|
|
*Symptoms:*
|
|
|
|
* When you use the {ref}/cluster-nodes-info.html[nodes info API] to retrieve
|
|
settings for a node, some information is missing.
|
|
|
|
*Resolution:*
|
|
|
|
This is intentional. Some of the settings are considered to be highly
|
|
sensitive: all `ssl` settings, ldap `bind_dn`, and `bind_password`.
|
|
For this reason, we filter these settings and do not expose them via
|
|
the nodes info API rest endpoint. You can also define additional
|
|
sensitive settings that should be hidden using the
|
|
`xpack.security.hide_settings` setting. For example, this snippet
|
|
hides the `url` settings of the `ldap1` realm and all settings of the
|
|
`ad1` realm.
|
|
|
|
[source, yaml]
|
|
------------------------------------------
|
|
xpack.security.hide_settings: xpack.security.authc.realms.ldap1.url,
|
|
xpack.security.authc.realms.ad1.*
|
|
------------------------------------------
|
|
|
|
[[security-trb-roles]]
|
|
=== Authorization exceptions
|
|
|
|
*Symptoms:*
|
|
|
|
* I configured the appropriate roles and the users, but I still get an
|
|
authorization exception.
|
|
* I can authenticate to LDAP, but I still get an authorization exception.
|
|
|
|
|
|
*Resolution:*
|
|
|
|
. Verify that the role names associated with the users match the roles defined
|
|
in the `roles.yml` file. You can use the `users` tool to list all the users. Any
|
|
unknown roles are marked with `*`.
|
|
+
|
|
--
|
|
[source, shell]
|
|
------------------------------------------
|
|
bin/xpack/users list
|
|
rdeniro : admin
|
|
alpacino : power_user
|
|
jacknich : monitoring,unknown_role* <1>
|
|
------------------------------------------
|
|
<1> `unknown_role` was not found in `roles.yml`
|
|
|
|
For more information about this command, see
|
|
{ref}/users-command.html[Users Command].
|
|
--
|
|
|
|
. If you are authenticating to LDAP, a number of configuration options can cause
|
|
this error.
|
|
+
|
|
--
|
|
|======================
|
|
|_group identification_ |
|
|
|
|
Groups are located by either an LDAP search or by the "memberOf" attribute on
|
|
the user. Also, If subtree search is turned off, it will search only one
|
|
level deep. See the <<ldap-settings, LDAP Settings>> for all the options.
|
|
There are many options here and sticking to the defaults will not work for all
|
|
scenarios.
|
|
|
|
| _group to role mapping_|
|
|
|
|
Either the `role_mapping.yml` file or the location for this file could be
|
|
misconfigured. See <<security-files, Security Files>> for more.
|
|
|
|
|_role definition_|
|
|
|
|
The role definition might be missing or invalid.
|
|
|
|
|======================
|
|
|
|
To help track down these possibilities, add the following lines to the end of
|
|
the `log4j2.properties` configuration file in the `CONFIG_DIR`:
|
|
|
|
[source,properties]
|
|
----------------
|
|
logger.authc.name = org.elasticsearch.xpack.security.authc
|
|
logger.authc.level = DEBUG
|
|
----------------
|
|
|
|
A successful authentication should produce debug statements that list groups and
|
|
role mappings.
|
|
--
|
|
|
|
[[security-trb-extraargs]]
|
|
=== Users command fails due to extra arguments
|
|
|
|
*Symptoms:*
|
|
|
|
* The `users` command fails with the following message:
|
|
`ERROR: extra arguments [...] were provided`.
|
|
|
|
*Resolution:*
|
|
|
|
This error occurs when the `users` tool is parsing the input and finds
|
|
unexpected arguments. This can happen when there are special characters used in
|
|
some of the arguments. For example, on Windows systems the `,` character is
|
|
considered a parameter separator; in other words `-r role1,role2` is translated
|
|
to `-r role1 role2` and the `users` tool only recognizes `role1` as an expected
|
|
parameter. The solution here is to quote the parameter: `-r "role1,role2"`.
|
|
|
|
For more information about this command, see
|
|
{ref}/users-command.html[Users Command].
|
|
|
|
[[trouble-shoot-active-directory]]
|
|
=== Users are frequently locked out of Active Directory
|
|
|
|
*Symptoms:*
|
|
|
|
* Certain users are being frequently locked out of Active Directory.
|
|
|
|
*Resolution:*
|
|
|
|
Check your realm configuration; realms are checked serially, one after another.
|
|
If your Active Directory realm is being checked before other realms and there
|
|
are usernames that appear in both Active Directory and another realm, a valid
|
|
login for one realm might be causing failed login attempts in another realm.
|
|
|
|
For example, if `UserA` exists in both Active Directory and a file realm, and
|
|
the Active Directory realm is checked first and file is checked second, an
|
|
attempt to authenticate as `UserA` in the file realm would first attempt to
|
|
authenticate against Active Directory and fail, before successfully
|
|
authenticating against the `file` realm. Because authentication is verified on
|
|
each request, the Active Directory realm would be checked - and fail - on each
|
|
request for `UserA` in the `file` realm. In this case, while the authentication
|
|
request completed successfully, the account on Active Directory would have
|
|
received several failed login attempts, and that account might become
|
|
temporarily locked out. Plan the order of your realms accordingly.
|
|
|
|
Also note that it is not typically necessary to define multiple Active Directory
|
|
realms to handle domain controller failures. When using Microsoft DNS, the DNS
|
|
entry for the domain should always point to an available domain controller.
|
|
|
|
|
|
[[trb-security-maccurl]]
|
|
=== Certificate verification fails for curl on Mac
|
|
|
|
*Symptoms:*
|
|
|
|
* `curl` on the Mac returns a certificate verification error even when the
|
|
`--cacert` option is used.
|
|
|
|
|
|
*Resolution:*
|
|
|
|
Apple's integration of `curl` with their keychain technology disables the
|
|
`--cacert` option.
|
|
See http://curl.haxx.se/mail/archive-2013-10/0036.html for more information.
|
|
|
|
You can use another tool, such as `wget`, to test certificates. Alternately, you
|
|
can add the certificate for the signing certificate authority MacOS system
|
|
keychain, using a procedure similar to the one detailed at the
|
|
http://support.apple.com/kb/PH14003[Apple knowledge base]. Be sure to add the
|
|
signing CA's certificate and not the server's certificate.
|
|
|
|
|
|
[[trb-security-sslhandshake]]
|
|
=== SSLHandshakeException causes connections to fail
|
|
|
|
*Symptoms:*
|
|
|
|
* A `SSLHandshakeException` causes a connection to a node to fail and indicates
|
|
that there is a configuration issue. Some of the common exceptions are shown
|
|
below with tips on how to resolve these issues.
|
|
|
|
|
|
*Resolution:*
|
|
|
|
`java.security.cert.CertificateException: No name matching node01.example.com found`::
|
|
+
|
|
--
|
|
Indicates that a client connection was made to `node01.example.com` but the
|
|
certificate returned did not contain the name `node01.example.com`. In most
|
|
cases, the issue can be resolved by ensuring the name is specified during
|
|
certificate creation. For more information, see <<ssl-tls>>. Another scenario is
|
|
when the environment does not wish to use DNS names in certificates at all. In
|
|
this scenario, all settings in `elasticsearch.yml` should only use IP addresses
|
|
including the `network.publish_host` setting.
|
|
--
|
|
|
|
`java.security.cert.CertificateException: No subject alternative names present`::
|
|
+
|
|
--
|
|
Indicates that a client connection was made to an IP address but the returned
|
|
certificate did not contain any `SubjectAlternativeName` entries. IP addresses
|
|
are only used for hostname verification if they are specified as a
|
|
`SubjectAlternativeName` during certificate creation. If the intent was to use
|
|
IP addresses for hostname verification, then the certificate will need to be
|
|
regenerated with the appropriate IP address. See <<ssl-tls>>.
|
|
--
|
|
|
|
`javax.net.ssl.SSLHandshakeException: null cert chain` and `javax.net.ssl.SSLException: Received fatal alert: bad_certificate`::
|
|
+
|
|
--
|
|
The `SSLHandshakeException` indicates that a self-signed certificate was
|
|
returned by the client that is not trusted as it cannot be found in the
|
|
`truststore` or `keystore`. This `SSLException` is seen on the client side of
|
|
the connection.
|
|
--
|
|
|
|
`sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` and `javax.net.ssl.SSLException: Received fatal alert: certificate_unknown`::
|
|
+
|
|
--
|
|
This `SunCertPathBuilderException` indicates that a certificate was returned
|
|
during the handshake that is not trusted. This message is seen on the client
|
|
side of the connection. The `SSLException` is seen on the server side of the
|
|
connection. The CA certificate that signed the returned certificate was not
|
|
found in the `keystore` or `truststore` and needs to be added to trust this
|
|
certificate.
|
|
--
|
|
|
|
[[trb-security-ssl]]
|
|
=== Common SSL/TLS exceptions
|
|
|
|
*Symptoms:*
|
|
|
|
* You might see some exceptions related to SSL/TLS in your logs. Some of the
|
|
common exceptions are shown below with tips on how to resolve these issues. +
|
|
|
|
|
|
|
|
*Resolution:*
|
|
|
|
`WARN: received plaintext http traffic on a https channel, closing connection`::
|
|
+
|
|
--
|
|
Indicates that there was an incoming plaintext http request. This typically
|
|
occurs when an external applications attempts to make an unencrypted call to the
|
|
REST interface. Please ensure that all applications are using `https` when
|
|
calling the REST interface with SSL enabled.
|
|
--
|
|
|
|
`org.elasticsearch.common.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:`::
|
|
+
|
|
--
|
|
Indicates that there was incoming plaintext traffic on an SSL connection. This
|
|
typically occurs when a node is not configured to use encrypted communication
|
|
and tries to connect to nodes that are using encrypted communication. Please
|
|
verify that all nodes are using the same setting for
|
|
`xpack.security.transport.ssl.enabled`.
|
|
|
|
For more information about this setting, see
|
|
{ref}/security-settings.html[Security Settings in {es}].
|
|
--
|
|
|
|
`java.io.StreamCorruptedException: invalid internal transport message format, got`::
|
|
+
|
|
--
|
|
Indicates an issue with data received on the transport interface in an unknown
|
|
format. This can happen when a node with encrypted communication enabled
|
|
connects to a node that has encrypted communication disabled. Please verify that
|
|
all nodes are using the same setting for `xpack.security.transport.ssl.enabled`.
|
|
|
|
For more information about this setting, see
|
|
{ref}/security-settings.html[Security Settings in {es}].
|
|
--
|
|
|
|
`java.lang.IllegalArgumentException: empty text`::
|
|
+
|
|
--
|
|
This exception is typically seen when a `https` request is made to a node that
|
|
is not using `https`. If `https` is desired, please ensure the following setting
|
|
is in `elasticsearch.yml`:
|
|
|
|
[source,yaml]
|
|
----------------
|
|
xpack.security.http.ssl.enabled: true
|
|
----------------
|
|
|
|
For more information about this setting, see
|
|
{ref}/security-settings.html[Security Settings in {es}].
|
|
--
|
|
|
|
`ERROR: unsupported ciphers [...] were requested but cannot be used in this JVM`::
|
|
+
|
|
--
|
|
This error occurs when a SSL/TLS cipher suite is specified that cannot supported
|
|
by the JVM that {es} is running in. Security tries to use the specified cipher
|
|
suites that are supported by this JVM. This error can occur when using the
|
|
Security defaults as some distributions of OpenJDK do not enable the PKCS11
|
|
provider by default. In this case, we recommend consulting your JVM
|
|
documentation for details on how to enable the PKCS11 provider.
|
|
|
|
Another common source of this error is requesting cipher suites that use
|
|
encrypting with a key length greater than 128 bits when running on an Oracle JDK.
|
|
In this case, you must install the
|
|
<<ciphers, JCE Unlimited Strength Jurisdiction Policy Files>>.
|
|
--
|
|
|
|
[[trb-security-internalserver]]
|
|
=== Internal Server Error in Kibana
|
|
|
|
*Symptoms:*
|
|
|
|
* In 5.1.1, an `UnhandledPromiseRejectionWarning` occurs and {kib} displays an
|
|
Internal Server Error.
|
|
//TBD: Is the same true for later releases?
|
|
|
|
*Resolution:*
|
|
|
|
If the Security plugin is enabled in {es} but disabled in {kib}, you must
|
|
still set `elasticsearch.username` and `elasticsearch.password` in `kibana.yml`.
|
|
Otherwise, {kib} cannot connect to {es}.
|
|
|
|
|
|
[[trb-security-setup]]
|
|
=== Setup-passwords command fails due to connection failure
|
|
|
|
The {ref}/setup-passwords.html[setup-passwords command] sets passwords for
|
|
the built-in users by sending user management API requests. If your cluster uses
|
|
SSL/TLS for the HTTP (REST) interface, the command attempts to establish a
|
|
connection with the HTTPS protocol. If the connection attempt fails, the
|
|
command fails.
|
|
|
|
*Symptoms:*
|
|
|
|
. {es} is running HTTPS, but the command fails to detect it and returns the
|
|
following errors:
|
|
+
|
|
--
|
|
[source, shell]
|
|
------------------------------------------
|
|
Cannot connect to elasticsearch node.
|
|
java.net.SocketException: Unexpected end of file from server
|
|
...
|
|
ERROR: Failed to connect to elasticsearch at
|
|
http://127.0.0.1:9200/_xpack/security/_authenticate?pretty.
|
|
Is the URL correct and elasticsearch running?
|
|
------------------------------------------
|
|
--
|
|
|
|
. SSL/TLS is configured, but trust cannot be established. The command returns
|
|
the following errors:
|
|
+
|
|
--
|
|
[source, shell]
|
|
------------------------------------------
|
|
SSL connection to
|
|
https://127.0.0.1:9200/_xpack/security/_authenticate?pretty
|
|
failed: sun.security.validator.ValidatorException:
|
|
PKIX path building failed:
|
|
sun.security.provider.certpath.SunCertPathBuilderException:
|
|
unable to find valid certification path to requested target
|
|
Please check the elasticsearch SSL settings under
|
|
xpack.security.http.ssl.
|
|
...
|
|
ERROR: Failed to establish SSL connection to elasticsearch at
|
|
https://127.0.0.1:9200/_xpack/security/_authenticate?pretty.
|
|
------------------------------------------
|
|
--
|
|
|
|
. The command fails because hostname verification fails, which results in the
|
|
following errors:
|
|
+
|
|
--
|
|
[source, shell]
|
|
------------------------------------------
|
|
SSL connection to
|
|
https://idp.localhost.test:9200/_xpack/security/_authenticate?pretty
|
|
failed: java.security.cert.CertificateException:
|
|
No subject alternative DNS name matching
|
|
elasticsearch.example.com found.
|
|
Please check the elasticsearch SSL settings under
|
|
xpack.security.http.ssl.
|
|
...
|
|
ERROR: Failed to establish SSL connection to elasticsearch at
|
|
https://elasticsearch.example.com:9200/_xpack/security/_authenticate?pretty.
|
|
------------------------------------------
|
|
--
|
|
|
|
*Resolution:*
|
|
|
|
. If your cluster uses TLS/SSL for the HTTP interface but the `setup-passwords`
|
|
command attempts to establish a non-secure connection, use the `--url` command
|
|
option to explicitly specify an HTTPS URL. Alternatively, set the
|
|
`xpack.security.http.ssl.enabled` setting to `true`.
|
|
|
|
. If the command does not trust the {es} server, verify that you configured the
|
|
`xpack.security.http.ssl.certificate_authorities` setting or the
|
|
`xpack.security.http.ssl.truststore.path` setting.
|
|
|
|
. If hostname verification fails, you can disable this verification by setting
|
|
`xpack.security.http.ssl.verification_mode` to `certificate`.
|
|
|
|
For more information about these settings, see
|
|
{ref}/security-settings.html[Security Settings in {es}].
|