OpenSearch/shield/docs/public/troubleshooting.asciidoc

236 lines
11 KiB
Plaintext
Raw Normal View History

[[troubleshooting]]
== Troubleshooting
[float]
=== `settings`
Some settings are not returned via the nodes settings API::
+
--
This is intentional. Some of the settings are considered to be highly sensitive (e.g. all `ssl` settings, ldap `bind_dn`,
`bind_password` and `hostname_verification`). For this reason, we filter these settings and not exposing them via the
nodes info API rest endpoint. It is also possible to define additional sensitive settings that should be hidden using
the `shield.hide_settings` setting:
[source, yaml]
------------------------------------------
shield.hide_settings: shield.authc.realms.ldap1.url, shield.authc.realms.ad1.*
------------------------------------------
The snippet above will also hide the `url` settings of the `ldap1` realm and all settings of the `ad1` realm.
--
[float]
=== `esusers`
I configured the appropriate roles and the users, but I still get an authorization exception::
+
--
Verify that the role names associated with the users match the roles defined in the `roles.yml` file. You
can use the `esusers` tool to list all the users. Any unknown roles are marked with `*`.
[source, shell]
------------------------------------------
esusers list
rdeniro : admin
alpacino : power_user
jacknich : marvel,unknown_role* <1>
------------------------------------------
<1> `unknown_role` was not found in `roles.yml`
--
ERROR: extra arguments [...] were provided::
+
--
This error occurs when the esusers tool is parsing the input and finds unexepected arguments. This can happen when there
are special characters used in some of the arguments. For example, on Windows systems the `,` character is considered
a parameter separator; in other words `-r role1,role2` is translated to `-r role1 role2` and the `esusers` tool only recognizes
`role1` as an expected parameter. The solution here is to quote the parameter: `-r "role1,role2"`.
--
[[trouble-shoot-active-directory]]
[float]
=== Active Directory
Certain users are being frequently locked out of Active Directory::
+
--
Check your realm configuration; realms are checked serially, one after another. If your Active Directory realm is being checked before other realms and there are usernames
that appear in both Active Directory and another realm, a valid login for one realm may be causing failed login attempts in another realm.
For example, if `UserA` exists in both Active Directory and esusers, and the Active Directory realm is checked first and
esusers is checked second, an attempt to authenticate as `UserA` in the esusers realm would first attempt to authenticate
against Active Directory and fail, before successfully authenticating against the esusers realm. Because authentication is
verified on each request, the Active Directory realm would be checked - and fail - on each request for `UserA` in the esusers
realm. In this case, while the Shield request completed successfully, the account on Active Directory would have received
several failed login attempts, and that account may become temporarily locked out. Plan the order of your realms accordingly.
Also note that it is not typically necessary to define multiple Active Directory realms to handle domain controller failures. When using Microsoft DNS, the DNS entry for
the domain should always point to an available domain controller.
--
[float]
=== LDAP
I can authenticate to LDAP, but I still get an authorization exception::
+
--
A number of configuration options can cause this error.
|======================
|_group identification_ |
Groups are located by either an LDAP search or by the "memberOf" attribute on
the user. Also, If subtree search is turned off, it will search only one
level deep. See the <<ldap-settings, LDAP Settings>> for all the options.
There are many options here and sticking to the defaults will not work for all
scenarios.
| _group to role mapping_|
Either the `role_mapping.yml` file or the location for this file could be
misconfigured. See <<ref-shield-files, Shield Files>> for more.
|_role definition_|
Either the `roles.yml` file or the location for this file could be
misconfigured. See <<ref-shield-files, Shield Files>> for more.
|======================
To help track down these possibilities, add `shield.authc: DEBUG` to the `logging.yml` configuration file in `CONFIG_DIR`.
A successful authentication should produce debug statements that list groups and role mappings.
--
[float]
=== Encryption & Certificates
`curl` on the Mac returns a certificate verification error even when the `--cacert` option is used::
+
--
Apple's integration of `curl` with their keychain technology disables the `--cacert` option.
See http://curl.haxx.se/mail/archive-2013-10/0036.html for more information.
You can use another tool, such as `wget`, to test certificates. Alternately, you can add the certificate for the
signing certificate authority MacOS system keychain, using a procedure similar to the one detailed at the
http://support.apple.com/kb/PH14003[Apple knowledge base]. Be sure to add the signing CA's certificate and not the server's certificate.
--
[float]
==== SSLHandshakeException causing connections to fail
A `SSLHandshakeException` will cause a connection to a node to fail and indicates that there is a configuration issue. Some of the
common exceptions are shown below with tips on how to resolve these issues.
`java.security.cert.CertificateException: No name matching node01.example.com found`::
+
--
Indicates that a client connection was made to `node01.example.com` but the certificate returned did not contain the name `node01.example.com`.
In most cases, the issue can be resolved by ensuring the name is specified as a `SubjectAlternativeName` during <<private-key, certificate creation>>.
Another scenario is when the environment does not wish to use DNS names in certificates at all. In this scenario, all settings
in `elasticsearch.yml` should only use IP addresses and the following setting needs to be set in `elasticsearch.yml`:
[source, yaml]
--------------------------------------------------
shield.ssl.hostname_verification.resolve_name: false
--------------------------------------------------
--
`java.security.cert.CertificateException: No subject alternative names present`::
+
--
Indicates that a client connection was made to an IP address but the returned certificate did not contain any `SubjectAlternativeName` entries.
IP addresses are only used for hostname verification if they are specified as a `SubjectAlternativeName` during
<<private-key, certificate creation>>. If the intent was to use IP addresses for hostname verification, then the certificate
will need to be regenerated. Also verify that `shield.ssl.hostname_verification.resolve_name: false` is *not* set in
`elasticsearch.yml`.
--
`javax.net.ssl.SSLHandshakeException: null cert chain` and `javax.net.ssl.SSLException: Received fatal alert: bad_certificate`::
+
--
The `SSLHandshakeException` above indicates that a self-signed certificate was returned by the client that is not trusted
as it cannot be found in the `truststore` or `keystore`. The `SSLException` above is seen on the client side of the connection.
--
`sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` and `javax.net.ssl.SSLException: Received fatal alert: certificate_unknown`::
+
--
The `SunCertPathBuilderException` above indicates that a certificate was returned during the handshake that is not trusted.
This message is seen on the client side of the connection. The `SSLException` above is seen on the server side of the
connection. The CA certificate that signed the returned certificate was not found in the `keystore` or `truststore` and
needs to be added to trust this certificate.
--
[float]
==== Other SSL/TLS related exceptions
The are other exceptions related to SSL that may be seen in the logs. Below you will find some common exceptions and their
meaning.
WARN: received plaintext http traffic on a https channel, closing connection::
+
--
Indicates that there was an incoming plaintext http request. This typically occurs when an external applications attempts
to make an unencrypted call to the REST interface. Please ensure that all applications are using `https` when calling the
REST interface with SSL enabled.
--
`org.elasticsearch.common.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:`::
+
--
Indicates that there was incoming plaintext traffic on an SSL connection. This typically occurs when a node is not
configured to use encrypted communication and tries to connect to nodes that are using encrypted communication. Please
verify that all nodes are using the same setting for `shield.transport.ssl`.
--
`java.io.StreamCorruptedException: invalid internal transport message format, got`::
+
--
Indicates an issue with data received on the transport interface in an unknown format. This can happen when a node with
encrypted communication enabled connects to a node that has encrypted communication disabled. Please verify that all
nodes are using the same setting for `shield.transport.ssl`.
--
`java.lang.IllegalArgumentException: empty text`::
+
--
The exception is typically seen when a `https` request is made to a node that is not using `https`. If `https` is desired,
please ensure the following setting is in `elasticsearch.yml`:
[source,yaml]
----------------
shield.http.ssl: true
----------------
--
ERROR: unsupported ciphers [...] were requested but cannot be used in this JVM::
+
--
This error occurs when a SSL/TLS cipher suite is specified that cannot supported by the JVM that Elasticsearch is running
in. Shield will try to use the specified cipher suites that are supported by this JVM. This error can occur when using
the Shield defaults as some distributions of OpenJDK do not enable the PKCS11 provider by default. In this case, we
recommend consulting your JVM documentation for details on how to enable the PKCS11 provider.
Another common source of this error is requesting cipher suites that use encrypting with a key length greater than 128 bits
when running on an Oracle JDK. In this case, you will need to install the <<ciphers, JCE Unlimited Strength Jurisdiction Policy Files>>.
--
[float]
=== Exceptions when unlicensed
WARN: Failed to execute IndicesStatsAction for ClusterInfoUpdateJob::
+
--
This warning occurs in the logs every 30 seconds when the Shield license is expired or invalid. It is caused by a periodic
internal request to gather disk usage information from the nodes and indices, to enable {ref}/disk.html[disk-based shard allocation].
Disk-based shard allocation is not required, though it is enabled by default.
If you are using Elasticsearch 1.4.3 or higher with disk-based shard allocation enabled, it will be automatically disabled when the Shield
license is expired or invalid, and will be automatically re-enabled when a valid Shield license is installed.
If you are using Elasticsearch 1.4.2 with disk-based shard allocation enabled, we recommend manually disabling disk-based shard
allocation while your Shield license is expired, and re-enabling it after installing a valid Shield license. Instructions for
disabling disk-based shard allocation are {ref}/disk.html[here].
--