236 lines
11 KiB
Plaintext
236 lines
11 KiB
Plaintext
[[troubleshooting]]
|
|
== Troubleshooting
|
|
|
|
[float]
|
|
=== `settings`
|
|
|
|
Some settings are not returned via the nodes settings API::
|
|
+
|
|
--
|
|
This is intentional. Some of the settings are considered to be highly sensitive (e.g. all `ssl` settings, ldap `bind_dn`,
|
|
`bind_password` and `hostname_verification`). For this reason, we filter these settings and not exposing them via the
|
|
nodes info API rest endpoint. It is also possible to define additional sensitive settings that should be hidden using
|
|
the `shield.hide_settings` setting:
|
|
|
|
[source, yaml]
|
|
------------------------------------------
|
|
shield.hide_settings: shield.authc.realms.ldap1.url, shield.authc.realms.ad1.*
|
|
------------------------------------------
|
|
|
|
The snippet above will also hide the `url` settings of the `ldap1` realm and all settings of the `ad1` realm.
|
|
--
|
|
|
|
[float]
|
|
=== `esusers`
|
|
|
|
I configured the appropriate roles and the users, but I still get an authorization exception::
|
|
+
|
|
--
|
|
Verify that the role names associated with the users match the roles defined in the `roles.yml` file. You
|
|
can use the `esusers` tool to list all the users. Any unknown roles are marked with `*`.
|
|
|
|
[source, shell]
|
|
------------------------------------------
|
|
esusers list
|
|
rdeniro : admin
|
|
alpacino : power_user
|
|
jacknich : marvel,unknown_role* <1>
|
|
------------------------------------------
|
|
|
|
<1> `unknown_role` was not found in `roles.yml`
|
|
--
|
|
|
|
ERROR: extra arguments [...] were provided::
|
|
+
|
|
--
|
|
This error occurs when the esusers tool is parsing the input and finds unexepected arguments. This can happen when there
|
|
are special characters used in some of the arguments. For example, on Windows systems the `,` character is considered
|
|
a parameter separator; in other words `-r role1,role2` is translated to `-r role1 role2` and the `esusers` tool only recognizes
|
|
`role1` as an expected parameter. The solution here is to quote the parameter: `-r "role1,role2"`.
|
|
--
|
|
|
|
[[trouble-shoot-active-directory]]
|
|
[float]
|
|
=== Active Directory
|
|
|
|
Certain users are being frequently locked out of Active Directory::
|
|
+
|
|
--
|
|
Check your realm configuration; realms are checked serially, one after another. If your Active Directory realm is being checked before other realms and there are usernames
|
|
that appear in both Active Directory and another realm, a valid login for one realm may be causing failed login attempts in another realm.
|
|
|
|
For example, if `UserA` exists in both Active Directory and esusers, and the Active Directory realm is checked first and
|
|
esusers is checked second, an attempt to authenticate as `UserA` in the esusers realm would first attempt to authenticate
|
|
against Active Directory and fail, before successfully authenticating against the esusers realm. Because authentication is
|
|
verified on each request, the Active Directory realm would be checked - and fail - on each request for `UserA` in the esusers
|
|
realm. In this case, while the Shield request completed successfully, the account on Active Directory would have received
|
|
several failed login attempts, and that account may become temporarily locked out. Plan the order of your realms accordingly.
|
|
|
|
Also note that it is not typically necessary to define multiple Active Directory realms to handle domain controller failures. When using Microsoft DNS, the DNS entry for
|
|
the domain should always point to an available domain controller.
|
|
--
|
|
|
|
[float]
|
|
=== LDAP
|
|
|
|
I can authenticate to LDAP, but I still get an authorization exception::
|
|
+
|
|
--
|
|
A number of configuration options can cause this error.
|
|
|
|
|======================
|
|
|_group identification_ |
|
|
|
|
Groups are located by either an LDAP search or by the "memberOf" attribute on
|
|
the user. Also, If subtree search is turned off, it will search only one
|
|
level deep. See the <<ldap-settings, LDAP Settings>> for all the options.
|
|
There are many options here and sticking to the defaults will not work for all
|
|
scenarios.
|
|
|
|
| _group to role mapping_|
|
|
|
|
Either the `role_mapping.yml` file or the location for this file could be
|
|
misconfigured. See <<ref-shield-files, Shield Files>> for more.
|
|
|
|
|_role definition_|
|
|
|
|
Either the `roles.yml` file or the location for this file could be
|
|
misconfigured. See <<ref-shield-files, Shield Files>> for more.
|
|
|
|
|======================
|
|
|
|
To help track down these possibilities, add `shield.authc: DEBUG` to the `logging.yml` configuration file in `CONFIG_DIR`.
|
|
A successful authentication should produce debug statements that list groups and role mappings.
|
|
--
|
|
|
|
|
|
[float]
|
|
=== Encryption & Certificates
|
|
|
|
`curl` on the Mac returns a certificate verification error even when the `--cacert` option is used::
|
|
+
|
|
--
|
|
Apple's integration of `curl` with their keychain technology disables the `--cacert` option.
|
|
See http://curl.haxx.se/mail/archive-2013-10/0036.html for more information.
|
|
|
|
You can use another tool, such as `wget`, to test certificates. Alternately, you can add the certificate for the
|
|
signing certificate authority MacOS system keychain, using a procedure similar to the one detailed at the
|
|
http://support.apple.com/kb/PH14003[Apple knowledge base]. Be sure to add the signing CA's certificate and not the server's certificate.
|
|
--
|
|
|
|
[float]
|
|
==== SSLHandshakeException causing connections to fail
|
|
|
|
A `SSLHandshakeException` will cause a connection to a node to fail and indicates that there is a configuration issue. Some of the
|
|
common exceptions are shown below with tips on how to resolve these issues.
|
|
|
|
`java.security.cert.CertificateException: No name matching node01.example.com found`::
|
|
+
|
|
--
|
|
Indicates that a client connection was made to `node01.example.com` but the certificate returned did not contain the name `node01.example.com`.
|
|
In most cases, the issue can be resolved by ensuring the name is specified as a `SubjectAlternativeName` during <<private-key, certificate creation>>.
|
|
Another scenario is when the environment does not wish to use DNS names in certificates at all. In this scenario, all settings
|
|
in `elasticsearch.yml` should only use IP addresses and the following setting needs to be set in `elasticsearch.yml`:
|
|
[source, yaml]
|
|
--------------------------------------------------
|
|
shield.ssl.hostname_verification.resolve_name: false
|
|
--------------------------------------------------
|
|
--
|
|
|
|
`java.security.cert.CertificateException: No subject alternative names present`::
|
|
+
|
|
--
|
|
Indicates that a client connection was made to an IP address but the returned certificate did not contain any `SubjectAlternativeName` entries.
|
|
IP addresses are only used for hostname verification if they are specified as a `SubjectAlternativeName` during
|
|
<<private-key, certificate creation>>. If the intent was to use IP addresses for hostname verification, then the certificate
|
|
will need to be regenerated. Also verify that `shield.ssl.hostname_verification.resolve_name: false` is *not* set in
|
|
`elasticsearch.yml`.
|
|
--
|
|
|
|
`javax.net.ssl.SSLHandshakeException: null cert chain` and `javax.net.ssl.SSLException: Received fatal alert: bad_certificate`::
|
|
+
|
|
--
|
|
The `SSLHandshakeException` above indicates that a self-signed certificate was returned by the client that is not trusted
|
|
as it cannot be found in the `truststore` or `keystore`. The `SSLException` above is seen on the client side of the connection.
|
|
--
|
|
|
|
`sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` and `javax.net.ssl.SSLException: Received fatal alert: certificate_unknown`::
|
|
+
|
|
--
|
|
The `SunCertPathBuilderException` above indicates that a certificate was returned during the handshake that is not trusted.
|
|
This message is seen on the client side of the connection. The `SSLException` above is seen on the server side of the
|
|
connection. The CA certificate that signed the returned certificate was not found in the `keystore` or `truststore` and
|
|
needs to be added to trust this certificate.
|
|
--
|
|
|
|
[float]
|
|
==== Other SSL/TLS related exceptions
|
|
|
|
The are other exceptions related to SSL that may be seen in the logs. Below you will find some common exceptions and their
|
|
meaning.
|
|
|
|
WARN: received plaintext http traffic on a https channel, closing connection::
|
|
+
|
|
--
|
|
Indicates that there was an incoming plaintext http request. This typically occurs when an external applications attempts
|
|
to make an unencrypted call to the REST interface. Please ensure that all applications are using `https` when calling the
|
|
REST interface with SSL enabled.
|
|
--
|
|
|
|
`org.elasticsearch.common.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:`::
|
|
+
|
|
--
|
|
Indicates that there was incoming plaintext traffic on an SSL connection. This typically occurs when a node is not
|
|
configured to use encrypted communication and tries to connect to nodes that are using encrypted communication. Please
|
|
verify that all nodes are using the same setting for `shield.transport.ssl`.
|
|
--
|
|
|
|
`java.io.StreamCorruptedException: invalid internal transport message format, got`::
|
|
+
|
|
--
|
|
Indicates an issue with data received on the transport interface in an unknown format. This can happen when a node with
|
|
encrypted communication enabled connects to a node that has encrypted communication disabled. Please verify that all
|
|
nodes are using the same setting for `shield.transport.ssl`.
|
|
--
|
|
|
|
`java.lang.IllegalArgumentException: empty text`::
|
|
+
|
|
--
|
|
The exception is typically seen when a `https` request is made to a node that is not using `https`. If `https` is desired,
|
|
please ensure the following setting is in `elasticsearch.yml`:
|
|
|
|
[source,yaml]
|
|
----------------
|
|
shield.http.ssl: true
|
|
----------------
|
|
--
|
|
|
|
ERROR: unsupported ciphers [...] were requested but cannot be used in this JVM::
|
|
+
|
|
--
|
|
This error occurs when a SSL/TLS cipher suite is specified that cannot supported by the JVM that Elasticsearch is running
|
|
in. Shield will try to use the specified cipher suites that are supported by this JVM. This error can occur when using
|
|
the Shield defaults as some distributions of OpenJDK do not enable the PKCS11 provider by default. In this case, we
|
|
recommend consulting your JVM documentation for details on how to enable the PKCS11 provider.
|
|
|
|
Another common source of this error is requesting cipher suites that use encrypting with a key length greater than 128 bits
|
|
when running on an Oracle JDK. In this case, you will need to install the <<ciphers, JCE Unlimited Strength Jurisdiction Policy Files>>.
|
|
--
|
|
|
|
[float]
|
|
=== Exceptions when unlicensed
|
|
|
|
WARN: Failed to execute IndicesStatsAction for ClusterInfoUpdateJob::
|
|
+
|
|
--
|
|
This warning occurs in the logs every 30 seconds when the Shield license is expired or invalid. It is caused by a periodic
|
|
internal request to gather disk usage information from the nodes and indices, to enable {ref}/disk.html[disk-based shard allocation].
|
|
Disk-based shard allocation is not required, though it is enabled by default.
|
|
|
|
If you are using Elasticsearch 1.4.3 or higher with disk-based shard allocation enabled, it will be automatically disabled when the Shield
|
|
license is expired or invalid, and will be automatically re-enabled when a valid Shield license is installed.
|
|
|
|
If you are using Elasticsearch 1.4.2 with disk-based shard allocation enabled, we recommend manually disabling disk-based shard
|
|
allocation while your Shield license is expired, and re-enabling it after installing a valid Shield license. Instructions for
|
|
disabling disk-based shard allocation are {ref}/disk.html[here].
|
|
-- |