honeymoose/OpenSearch
1
0
Fork 0
mirror of https://github.com/honeymoose/OpenSearch.git synced 2025-02-06 04:58:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove index audit output type (#37707)

Browse Source 
This commit removes the Index Audit Output type, following its deprecation
in 6.7 by 8765a31d4e6770. It also adds the migration notice (settings notice).

In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the
`xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881
This commit is contained in:
Albert Zaharovits 2019-01-24 12:36:10 +02:00 committed by GitHub
parent f12bfb4684
commit b6936e3c1e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
26 changed files with 51 additions and 3874 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/reference/migration/migrate_7_0/settings.asciidoc
View File

@ -154,3 +154,18 @@ node's keystore, i.e., they are not to be specified via the cluster settings API
`xpack.notification.pagerduty.account.<id>.secure_service_api_key`
- `xpack.notification.slack.account.<id>.url`, instead use
`xpack.notification.slack.account.<id>.secure_url`
[float]
[[remove-audit-index-output]]
==== Audit index output type removed
All the settings under the `xpack.security.audit.index` namespace have been
removed. In addition, the `xpack.security.audit.outputs` setting has been
removed as well.
These settings enabled and configured the audit index output type. This output
type has been removed because it was unreliable in certain scenarios and this
could have lead to dropping audit events while the operations on the system
were allowed to continue as usual. The recommended replacement is the
use of the `logfile` audit output type and using other components from the
Elastic Stack to handle the indexing part.

126
docs/reference/settings/audit-settings.asciidoc
View File

@ -14,38 +14,19 @@ file. For more information, see
`xpack.security.audit.enabled`::
Set to `true` to enable auditing on the node. The default value is `false`.
`xpack.security.audit.outputs`::
Specifies where audit logs are output. For example: `[ index, logfile ]`. The
default value is `logfile`, which puts the auditing events in a dedicated
file named `<clustername>_audit.log` on each node.
You can also specify `index`, which puts the auditing events in an {es} index
that is prefixed with `.security_audit_log`. The index can reside on the same
cluster or a separate cluster.
+
For backwards compatibility reasons, if you use the logfile output type, a
`<clustername>_access.log` file is also created. It contains the same
information, but it uses the older (pre-6.5.0) formatting style.
If the backwards compatible format is not required, it should be disabled.
To do that, change its logger level to `off` in the `log4j2.properties` file.
For more information, see <<configuring-logging-levels>>.
+
--
TIP: If the index is unavailable, it is possible for auditing events to
be lost. The `index` output type should therefore be used in conjunction with
the `logfile` output type and the latter should be the official record of events.
--
This puts the auditing events in a dedicated file named `<clustername>_audit.log`
on each node. For more information, see <<configuring-logging-levels>>.
[[event-audit-settings]]
==== Audited Event Settings
The events and some other information about what gets logged can be
controlled by using the following settings:
The events and some other information about what gets logged can be controlled
by using the following settings:
`xpack.security.audit.logfile.events.include`::
Specifies which events to include in the auditing output. The default value is:
`access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted`.
`access_denied, access_granted, anonymous_access_denied, authentication_failed,
connection_denied, tampered_request, run_as_denied, run_as_granted`.
`xpack.security.audit.logfile.events.exclude`::
Excludes the specified events from the output. By default, no events are
@ -113,98 +94,3 @@ A list of index names or wildcards. The specified policy will
not print audit events when all the indices in the event match
these values. If the event concerns several indices, some of which are
*not* covered by the policy, the policy will *not* cover this event.
[[index-audit-settings]]
==== Audit Log Indexing Configuration Settings
`xpack.security.audit.index.bulk_size`::
Controls how many audit events are batched into a single write. The default
value is `1000`.
`xpack.security.audit.index.flush_interval`::
Controls how often buffered events are flushed to the index. The default value
is `1s`.
`xpack.security.audit.index.rollover`::
Controls how often to roll over to a new index: `hourly`, `daily`, `weekly`, or
`monthly`. The default value is `daily`.
`xpack.security.audit.index.events.include`::
Specifies the audit events to be indexed. The default value is
`anonymous_access_denied, authentication_failed, realm_authentication_failed, access_granted, access_denied, tampered_request, connection_granted, connection_denied, run_as_granted, run_as_denied`.
See {xpack-ref}/audit-event-types.html[Audit Entry Types] for the
complete list.
`xpack.security.audit.index.events.exclude`::
Excludes the specified auditing events from indexing. By default, no events are
excluded.
`xpack.security.audit.index.events.emit_request_body`::
Specifies whether to include the request body from REST requests on certain
event types such as `authentication_failed`. The default value is `false`.
`xpack.security.audit.index.settings`::
Specifies settings for the indices that the events are stored in. For example,
the following configuration sets the number of shards and replicas to 1 for the
audit indices:
+
--
[source,yaml]
----------------------------
xpack.security.audit.index.settings:
index:
number_of_shards: 1
number_of_replicas: 1
----------------------------
--
+
--
NOTE: These settings apply to the local audit indices, as well as to the
<<remote-audit-settings, remote audit indices>>, but only if the remote cluster
does *not* have {security-features} enabled or the {es} versions are different.
If the remote cluster has {security-features} enabled and the versions coincide,
the settings for the audit indices there will take precedence,
even if they are unspecified (i.e. left to defaults).
--
[[remote-audit-settings]]
==== Remote Audit Log Indexing Configuration Settings
To index audit events to a remote {es} cluster, you configure the following
`xpack.security.audit.index.client` settings:
`xpack.security.audit.index.client.hosts`::
Specifies a comma-separated list of `host:port` pairs. These hosts should be
nodes in the remote cluster. If you are using default values for the
<<common-network-settings,`transport.port`>> setting, you can omit the
`port` value. Otherwise, it must match the `transport.port` setting.
`xpack.security.audit.index.client.cluster.name`::
Specifies the name of the remote cluster.
`xpack.security.audit.index.client.xpack.security.user`::
Specifies the `username:password` pair that is used to authenticate with the
remote cluster. This user must have authority to create the `.security-audit`
index on the remote cluster.
If the remote {es} cluster has Transport Layer Security (TLS/SSL) enabled, you
must set the following setting to `true`:
`xpack.security.audit.index.client.xpack.security.transport.ssl.enabled`::
Used to enable or disable TLS/SSL for the transport client that forwards audit
logs to the remote cluster. The default is `false`.
You must also specify the information necessary to access certificates. See
<<auditing-tls-ssl-settings>>.
You can pass additional settings to the remote client by specifying them in the
`xpack.security.audit.index.client` namespace. For example, you can add
<<modules-transport,transport settings>> and
<<tcp-settings,advanced TCP settings>> in that namespace. To allow the remote
client to discover all of the nodes in the remote cluster you can specify the
`client.transport.sniff` setting:
[source,yaml]
----------------------------
xpack.security.audit.index.client.transport.sniff: true
----------------------------

10
docs/reference/settings/security-settings.asciidoc
View File

@ -1303,16 +1303,6 @@ transport profile, use the prefix `transport.profiles.$PROFILE.xpack.security.`
append the portion of the setting after `xpack.security.transport.`. For the key
setting, this would be `transport.profiles.$PROFILE.xpack.security.ssl.key`.
[[auditing-tls-ssl-settings]]
:ssl-prefix: xpack.security.audit.index.client.xpack.security.transport
:component: Auditing
:client-auth-default!:
:server!:
include::ssl-settings.asciidoc[]
See also <<remote-audit-settings>>.
[float]
[[ip-filtering-settings]]
==== IP filtering settings

21
x-pack/docs/en/security/auditing/auditing-search-queries.asciidoc
View File

@ -14,25 +14,12 @@ audit events that are generated by the REST layer can be toggled to output
the request body to the audit log.
To make certain audit events include the request body, edit the following
settings in the `elasticsearch.yml` file:
setting in the `elasticsearch.yml` file:
* For the `logfile` audit output:
+
--
[source,yaml]
----------------------------
xpack.security.audit.logfile.events.emit_request_body: true
----------------------------
--
* For the `index` output:
+
--
[source,yaml]
----------------------------
xpack.security.audit.index.events.emit_request_body: true
----------------------------
--
IMPORTANT: No filtering is performed when auditing, so sensitive data might be
audited in plain text when audit events include the request body. Also, the
@ -44,10 +31,8 @@ generated in the REST layer and can access the request body. Most of them are no
included by default.
A good practical piece of advice is to add `authentication_success` to the event
types that are audited. Add it to the list in the
`xpack.security.audit.logfile.events.include` or
`xpack.security.audit.index.events.include` settings. This type is not audited
by default.
types that are audited (add it to the list in the `xpack.security.audit.logfile.events.include`),
as this event type is not audited by default.
NOTE: Typically, the include list contains other event types as well, such as
`access_granted` or `access_denied`.

9
x-pack/docs/en/security/auditing/event-types.asciidoc
View File

@ -210,8 +210,7 @@ that have been previously described:
=== Audit event attributes for the deprecated formats
The following table shows the common attributes that can be associated with
every event, when it is output to the `<clustername>_access.log` file or to the
<<audit-index, index>>.
every event, when it is output to the `<clustername>_access.log` file.
.Common attributes
[cols="2,7",options="header"]
@ -229,10 +228,8 @@ every event, when it is output to the `<clustername>_access.log` file or to the
`run_as_denied`, `run_as_granted`.
|======
For an event in the <<audit-log-output,audit log file output>>, these are
positional attributes, which are printed at the beginning of each log line and
are not adjoined by the attribute name. As a matter of course, the names are
present for each attribute when the event is forwarded to the <<audit-index, index audit output>>.
These are positional attributes, which are printed at the beginning of each log line and
are not adjoined by the attribute name.
The attribute `origin_address` is also common to every audit event. It is always
named, that is, it is not positional. It denotes the source IP address of the

99
x-pack/docs/en/security/auditing/forwarding-logs.asciidoc
View File

@ -1,99 +0,0 @@
[role="xpack"]
[[forwarding-audit-logfiles]]
=== Forwarding audit logs to a remote cluster
When you are auditing security events, you can optionally store the logs in an
{es} index on a remote cluster. The logs are sent to the remote cluster by
using the {javaclient}/transport-client.html[transport client].
. Configure auditing such that the logs are stored in {es} rolling indices.
See <<audit-index>>.
. Establish a connection to the remote cluster by configuring the following
`xpack.security.audit.index.client` settings:
+
--
[source, yaml]
--------------------------------------------------
xpack.security.audit.index.client.hosts: 192.168.0.1, 192.168.0.2 <1>
xpack.security.audit.index.client.cluster.name: logging-prod <2>
xpack.security.audit.index.client.xpack.security.user: myuser:mypassword <3>
--------------------------------------------------
<1> A list of hosts in the remote cluster. If you are not using the default
value for the `transport.port` setting on the remote cluster, you must
specify the appropriate port number (prefixed by a colon) after each host.
<2> The remote cluster name.
<3> A valid user and password, which must have authority to create the
`.security-audit` index on the remote cluster.
For more information about these settings, see
{ref}/auditing-settings.html#remote-audit-settings[Remote audit log indexing configuration settings].
--
. If the remote cluster has Transport Layer Security (TLS/SSL) enabled, you
must specify extra security settings:
.. {ref}/configuring-tls.html#node-certificates[Generate a node certificate on
the remote cluster], then copy that certificate to the client.
.. Enable TLS and specify the information required to access the node certificate.
*** If the signed certificate is in PKCS#12 format, add the following information
to the `elasticsearch.yml` file:
+
--
[source,yaml]
-----------------------------------------------------------
xpack.security.audit.index.client.xpack.security.transport.ssl.enabled: true
xpack.security.audit.index.client.xpack.security.transport.ssl.keystore.path: certs/remote-elastic-certificates.p12
xpack.security.audit.index.client.xpack.security.transport.ssl.truststore.path: certs/remote-elastic-certificates.p12
-----------------------------------------------------------
For more information about these settings, see
{ref}/security-settings.html#auditing-tls-ssl-settings[Auditing TLS settings].
--
*** If the certificate is in PEM format, add the following information to the
`elasticsearch.yml` file:
+
--
[source, yaml]
--------------------------------------------------
xpack.security.audit.index.client.xpack.security.transport.ssl.enabled: true
xpack.security.audit.index.client.xpack.security.transport.ssl.key: /home/es/config/audit-client.key
xpack.security.audit.index.client.xpack.security.transport.ssl.certificate: /home/es/config/audit-client.crt
xpack.security.audit.index.client.xpack.security.transport.ssl.certificate_authorities: [ "/home/es/config/remote-ca.crt" ]
--------------------------------------------------
For more information about these settings, see
{ref}/security-settings.html#auditing-tls-ssl-settings[Auditing TLS settings].
--
.. If you secured the certificate with a password, add the password to
your {es} keystore:
*** If the signed certificate is in PKCS#12 format, use the following commands:
+
--
[source,shell]
-----------------------------------------------------------
bin/elasticsearch-keystore add xpack.security.audit.index.client.xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.audit.index.client.xpack.security.transport.ssl.truststore.secure_password
-----------------------------------------------------------
--
*** If the certificate is in PEM format, use the following commands:
+
--
[source,shell]
-----------------------------------------------------------
bin/elasticsearch-keystore add xpack.security.audit.index.client.xpack.security.transport.ssl.secure_key_passphrase
-----------------------------------------------------------
--
. Restart {es}.
When these steps are complete, your audit logs are stored in {es} rolling
indices on the remote cluster.

6
x-pack/docs/en/security/auditing/index.asciidoc
View File

@ -8,11 +8,5 @@ include::event-types.asciidoc[]
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/output-logfile.asciidoc
include::output-logfile.asciidoc[]
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/output-index.asciidoc
include::output-index.asciidoc[]
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/auditing-search-queries.asciidoc
include::auditing-search-queries.asciidoc[]
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/forwarding-logs.asciidoc
include::forwarding-logs.asciidoc[]

49
x-pack/docs/en/security/auditing/output-index.asciidoc
View File

@ -1,49 +0,0 @@
[role="xpack"]
[[audit-index]]
=== Index audit output
In addition to logging to a file, you can store audit logs in Elasticsearch
rolling indices. These indices can be either on the same cluster, or on a
remote cluster. You configure the following settings in
`elasticsearch.yml` to control how audit entries are indexed. To enable
this output, you need to configure the setting `xpack.security.audit.outputs`
in the `elasticsearch.yml` file:
[source,yaml]
----------------------------
xpack.security.audit.outputs: [ index, logfile ]
----------------------------
For more configuration options, see
{ref}/auditing-settings.html#index-audit-settings[Audit log indexing configuration settings].
IMPORTANT: No filtering is performed when auditing, so sensitive data may be
audited in plain text when including the request body in audit events.
[float]
==== Audit index settings
You can also configure settings for the indices that the events are stored in.
These settings are configured in the `xpack.security.audit.index.settings` namespace
in `elasticsearch.yml`. For example, the following configuration sets the
number of shards and replicas to 1 for the audit indices:
[source,yaml]
----------------------------
xpack.security.audit.index.settings:
index:
number_of_shards: 1
number_of_replicas: 1
----------------------------
These settings apply to the local audit indices, as well as to the
<<forwarding-audit-logfiles, remote audit indices>>, but only if the remote cluster
does *not* have {security-features} enabled or the {es} versions are different.
If the remote cluster has {security-features} enabled and the versions coincide,
the settings for the audit indices there will take precedence,
even if they are unspecified (i.e. left to defaults).
NOTE: Audit events are batched for indexing so there is a lag before
events appear in the index. You can control how frequently batches of
events are pushed to the index by setting
`xpack.security.audit.index.flush_interval` in `elasticsearch.yml`.

25
x-pack/docs/en/security/auditing/overview.asciidoc
View File

@ -13,26 +13,5 @@ Audit logs are **disabled** by default. To enable this functionality, you
must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`.
============================================================================
The {es} {security-features} provide two ways to persist audit logs:
* The <<audit-log-output, `logfile`>> output, which persists events to
a dedicated `<clustername>_audit.log` file on the host's file system.
For backwards compatibility reasons, a file named `<clustername>_access.log`
is also generated.
* The <<audit-index, `index`>> output, which persists events to an Elasticsearch
index. The audit index can reside on the same cluster, or a separate cluster.
By default, only the `logfile` output is used when enabling auditing,
implicitly outputting to both `<clustername>_audit.log` and `<clustername>_access.log`.
To facilitate browsing and analyzing the events, you can also enable
indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`:
[source,yaml]
----------------------------
xpack.security.audit.outputs: [ index, logfile ]
----------------------------
TIP: If you choose to enable the `index` output type, we strongly recommend that
you still use the `logfile` output as the official record of events. If the
target index is unavailable (for example, during a rolling upgrade), the `index`
output can lose messages.
The audit log persists events to a dedicated `<clustername>_audit.log` file on
the host's file system (on each node).

5
x-pack/docs/en/security/configuring-es.asciidoc
View File

@ -131,9 +131,8 @@ and <<auditing-settings>>.
.. Restart {es}.
By default, events are logged to a dedicated `elasticsearch-access.log` file in
`ES_HOME/logs`. You can also store the events in an {es} index for
easier analysis and control what events are logged.
Events are logged to a dedicated `<clustername>_audit.log` file in
`ES_HOME/logs`, on each cluster node.
--
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/docs/reference/security/securing-communications/securing-elasticsearch.asciidoc

163
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java
View File

@ -23,15 +23,12 @@ import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.cluster.node.DiscoveryNodes;
import org.elasticsearch.cluster.service.ClusterService;
import org.elasticsearch.common.Booleans;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.inject.Module;
import org.elasticsearch.common.inject.util.Providers;
import org.elasticsearch.common.io.stream.NamedWriteableRegistry;
import org.elasticsearch.common.logging.LoggerMessageFormat;
import org.elasticsearch.common.network.NetworkModule;
import org.elasticsearch.common.network.NetworkService;
import org.elasticsearch.common.regex.Regex;
import org.elasticsearch.common.settings.ClusterSettings;
import org.elasticsearch.common.settings.IndexScopedSettings;
import org.elasticsearch.common.settings.Setting;
@ -42,12 +39,7 @@ import org.elasticsearch.common.util.BigArrays;
import org.elasticsearch.common.util.PageCacheRecycler;
import org.elasticsearch.common.util.concurrent.ThreadContext;
import org.elasticsearch.common.util.set.Sets;
import org.elasticsearch.common.xcontent.LoggingDeprecationHandler;
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
import org.elasticsearch.common.xcontent.XContent;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.discovery.DiscoveryModule;
import org.elasticsearch.env.Environment;
import org.elasticsearch.env.NodeEnvironment;
@ -126,7 +118,6 @@ import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions;
import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissionsCache;
import org.elasticsearch.xpack.core.security.authz.store.ReservedRolesStore;
import org.elasticsearch.xpack.core.security.authz.store.RoleRetrievalResult;
import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
import org.elasticsearch.xpack.core.security.support.Automatons;
import org.elasticsearch.xpack.core.security.user.AnonymousUser;
import org.elasticsearch.xpack.core.ssl.SSLConfiguration;
@ -136,7 +127,6 @@ import org.elasticsearch.xpack.core.ssl.TLSLicenseBootstrapCheck;
import org.elasticsearch.xpack.core.ssl.action.GetCertificateInfoAction;
import org.elasticsearch.xpack.core.ssl.action.TransportGetCertificateInfoAction;
import org.elasticsearch.xpack.core.ssl.rest.RestGetCertificateInfoAction;
import org.elasticsearch.xpack.core.template.TemplateUtils;
import org.elasticsearch.xpack.security.action.filter.SecurityActionFilter;
import org.elasticsearch.xpack.security.action.interceptor.BulkShardRequestInterceptor;
import org.elasticsearch.xpack.security.action.interceptor.IndicesAliasesRequestInterceptor;
@ -172,8 +162,6 @@ import org.elasticsearch.xpack.security.action.user.TransportPutUserAction;
import org.elasticsearch.xpack.security.action.user.TransportSetEnabledAction;
import org.elasticsearch.xpack.security.audit.AuditTrail;
import org.elasticsearch.xpack.security.audit.AuditTrailService;
import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail;
import org.elasticsearch.xpack.security.audit.index.IndexNameResolver;
import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail;
import org.elasticsearch.xpack.security.authc.AuthenticationService;
import org.elasticsearch.xpack.security.authc.InternalRealms;
@ -224,11 +212,7 @@ import org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServe
import org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport;
import org.elasticsearch.xpack.security.transport.nio.SecurityNioHttpServerTransport;
import org.elasticsearch.xpack.security.transport.nio.SecurityNioTransport;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.time.Clock;
import java.util.ArrayList;
@ -237,7 +221,6 @@ import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@ -262,14 +245,6 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
private static final Logger logger = LogManager.getLogger(Security.class);
static final Setting<List<String>> AUDIT_OUTPUTS_SETTING =
Setting.listSetting(SecurityField.setting("audit.outputs"),
Function.identity(),
s -> s.keySet().contains(SecurityField.setting("audit.outputs"))
? Collections.emptyList()
: Collections.singletonList(LoggingAuditTrail.NAME),
Property.NodeScope);
private final Settings settings;
private final Environment env;
private final boolean enabled;
@ -286,7 +261,6 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
private final SetOnce<TokenService> tokenService = new SetOnce<>();
private final SetOnce<SecurityActionFilter> securityActionFilter = new SetOnce<>();
private final SetOnce<SecurityIndexManager> securityIndex = new SetOnce<>();
private final SetOnce<IndexAuditTrail> indexAuditTrail = new SetOnce<>();
private final SetOnce<NioGroupFactory> groupFactory = new SetOnce<>();
private final List<BootstrapCheck> bootstrapChecks;
private final List<SecurityExtension> securityExtensions = new ArrayList<>();
@ -324,7 +298,6 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
}
private static void runStartupChecks(Settings settings) {
validateAutoCreateIndex(settings);
validateRealmSettings(settings);
}
@ -402,31 +375,11 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
securityContext.set(new SecurityContext(settings, threadPool.getThreadContext()));
components.add(securityContext.get());
// audit trails construction
Set<AuditTrail> auditTrails = new LinkedHashSet<>();
if (XPackSettings.AUDIT_ENABLED.get(settings)) {
List<String> outputs = AUDIT_OUTPUTS_SETTING.get(settings);
if (outputs.isEmpty()) {
throw new IllegalArgumentException("Audit logging is enabled but there are zero output types in "
+ XPackSettings.AUDIT_ENABLED.getKey());
}
for (String output : outputs) {
switch (output) {
case LoggingAuditTrail.NAME:
auditTrails.add(new LoggingAuditTrail(settings, clusterService, threadPool));
break;
case IndexAuditTrail.NAME:
indexAuditTrail.set(new IndexAuditTrail(settings, client, threadPool, clusterService));
auditTrails.add(indexAuditTrail.get());
break;
default:
throw new IllegalArgumentException("Unknown audit trail output [" + output + "]");
}
}
}
final AuditTrailService auditTrailService =
new AuditTrailService(new ArrayList<>(auditTrails), getLicenseState());
// audit trail service construction
final List<AuditTrail> auditTrails = XPackSettings.AUDIT_ENABLED.get(settings)
? Collections.singletonList(new LoggingAuditTrail(settings, clusterService, threadPool))
: Collections.emptyList();
final AuditTrailService auditTrailService = new AuditTrailService(auditTrails, getLicenseState());
components.add(auditTrailService);
this.auditTrailService.set(auditTrailService);
@ -613,9 +566,7 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
IPFilter.addSettings(settingsList);
// audit settings
settingsList.add(AUDIT_OUTPUTS_SETTING);
LoggingAuditTrail.registerSettings(settingsList);
IndexAuditTrail.registerSettings(settingsList);
// authentication and authorization settings
AnonymousUser.addSettings(settingsList);
@ -828,92 +779,6 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
}
}
static boolean indexAuditLoggingEnabled(Settings settings) {
if (XPackSettings.AUDIT_ENABLED.get(settings)) {
List<String> outputs = AUDIT_OUTPUTS_SETTING.get(settings);
for (String output : outputs) {
if (output.equals(IndexAuditTrail.NAME)) {
return true;
}
}
}
return false;
}
static void validateAutoCreateIndex(Settings settings) {
String value = settings.get("action.auto_create_index");
if (value == null) {
return;
}
final boolean indexAuditingEnabled = Security.indexAuditLoggingEnabled(settings);
if (indexAuditingEnabled) {
String auditIndex = IndexAuditTrailField.INDEX_NAME_PREFIX + "*";
String errorMessage = LoggerMessageFormat.format(
"the [action.auto_create_index] setting value [{}] is too" +
" restrictive. disable [action.auto_create_index] or set it to include " +
"[{}]", (Object) value, auditIndex);
if (Booleans.isFalse(value)) {
throw new IllegalArgumentException(errorMessage);
}
if (Booleans.isTrue(value)) {
return;
}
String[] matches = Strings.commaDelimitedListToStringArray(value);
List<String> indices = new ArrayList<>();
DateTime now = new DateTime(DateTimeZone.UTC);
// just use daily rollover
indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now, IndexNameResolver.Rollover.DAILY));
indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusDays(1),
IndexNameResolver.Rollover.DAILY));
indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(1),
IndexNameResolver.Rollover.DAILY));
indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(2),
IndexNameResolver.Rollover.DAILY));
indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(3),
IndexNameResolver.Rollover.DAILY));
indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(4),
IndexNameResolver.Rollover.DAILY));
indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(5),
IndexNameResolver.Rollover.DAILY));
indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(6),
IndexNameResolver.Rollover.DAILY));
for (String index : indices) {
boolean matched = false;
for (String match : matches) {
char c = match.charAt(0);
if (c == '-') {
if (Regex.simpleMatch(match.substring(1), index)) {
throw new IllegalArgumentException(errorMessage);
}
} else if (c == '+') {
if (Regex.simpleMatch(match.substring(1), index)) {
matched = true;
break;
}
} else {
if (Regex.simpleMatch(match, index)) {
matched = true;
break;
}
}
}
if (!matched) {
throw new IllegalArgumentException(errorMessage);
}
}
logger.warn("the [action.auto_create_index] setting is configured to be restrictive [{}]. " +
" for the next 6 months audit indices are allowed to be created, but please make sure" +
" that any future history indices after 6 months with the pattern " +
"[.security_audit_log*] are allowed to be created", value);
}
}
@Override
public List<TransportInterceptor> getTransportInterceptors(NamedWriteableRegistry namedWriteableRegistry, ThreadContext threadContext) {
if (transportClientMode || enabled == false) { // don't register anything if we are not enabled
@ -998,23 +863,9 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
@Override
public UnaryOperator<Map<String, IndexTemplateMetaData>> getIndexTemplateMetaDataUpgrader() {
return templates -> {
// .security index is not managed by using templates anymore
templates.remove(SECURITY_TEMPLATE_NAME);
final XContent xContent = XContentFactory.xContent(XContentType.JSON);
final byte[] auditTemplate = TemplateUtils.loadTemplate("/" + IndexAuditTrail.INDEX_TEMPLATE_NAME + ".json",
Version.CURRENT.toString(), SecurityIndexManager.TEMPLATE_VERSION_PATTERN).getBytes(StandardCharsets.UTF_8);
try (XContentParser parser = xContent
.createParser(NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE, auditTemplate)) {
IndexTemplateMetaData auditMetadata = new IndexTemplateMetaData.Builder(
IndexTemplateMetaData.Builder.fromXContent(parser, IndexAuditTrail.INDEX_TEMPLATE_NAME))
.settings(IndexAuditTrail.customAuditIndexSettings(settings, logger))
.build();
templates.put(IndexAuditTrail.INDEX_TEMPLATE_NAME, auditMetadata);
} catch (IOException e) {
// TODO: should we handle this with a thrown exception?
logger.error("Error loading template [{}] as part of metadata upgrading", IndexAuditTrail.INDEX_TEMPLATE_NAME);
}
templates.remove("security_audit_log");
return templates;
};
}

7
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/SecurityFeatureSet.java
View File

@ -16,11 +16,13 @@ import org.elasticsearch.xpack.core.XPackField;
import org.elasticsearch.xpack.core.XPackSettings;
import org.elasticsearch.xpack.core.security.SecurityFeatureSetUsage;
import org.elasticsearch.xpack.core.security.user.AnonymousUser;
import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail;
import org.elasticsearch.xpack.security.authc.Realms;
import org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore;
import org.elasticsearch.xpack.security.authz.store.CompositeRolesStore;
import org.elasticsearch.xpack.security.transport.filter.IPFilter;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
@ -153,7 +155,10 @@ public class SecurityFeatureSet implements XPackFeatureSet {
static Map<String, Object> auditUsage(Settings settings) {
Map<String, Object> map = new HashMap<>(2);
map.put("enabled", XPackSettings.AUDIT_ENABLED.get(settings));
map.put("outputs", Security.AUDIT_OUTPUTS_SETTING.get(settings));
if (XPackSettings.AUDIT_ENABLED.get(settings)) {
// the only available output type is "logfile", but the optputs=<list> is to keep compatibility with previous reporting format
map.put("outputs", Arrays.asList(LoggingAuditTrail.NAME));
}
return map;
}

1240
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/audit/index/IndexAuditTrail.java
View File

File diff suppressed because it is too large Load Diff

13
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecurityFeatureSetTests.java
View File

@ -21,6 +21,7 @@ import org.elasticsearch.xpack.core.XPackSettings;
import org.elasticsearch.xpack.core.security.SecurityFeatureSetUsage;
import org.elasticsearch.xpack.core.security.user.AnonymousUser;
import org.elasticsearch.xpack.core.watcher.support.xcontent.XContentSource;
import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail;
import org.elasticsearch.xpack.security.authc.Realms;
import org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore;
import org.elasticsearch.xpack.security.authz.store.CompositeRolesStore;
@ -97,12 +98,6 @@ public class SecurityFeatureSetTests extends ESTestCase {
settings.put("xpack.security.transport.ssl.enabled", transportSSLEnabled);
final boolean auditingEnabled = randomBoolean();
settings.put(XPackSettings.AUDIT_ENABLED.getKey(), auditingEnabled);
final String[] auditOutputs = randomFrom(
new String[] { "logfile" },
new String[] { "index" },
new String[] { "logfile", "index" }
);
settings.putList(Security.AUDIT_OUTPUTS_SETTING.getKey(), auditOutputs);
final boolean httpIpFilterEnabled = randomBoolean();
final boolean transportIPFilterEnabled = randomBoolean();
when(ipFilter.usageStats())
@ -192,7 +187,11 @@ public class SecurityFeatureSetTests extends ESTestCase {
// auditing
assertThat(source.getValue("audit.enabled"), is(auditingEnabled));
assertThat(source.getValue("audit.outputs"), contains(auditOutputs));
if (auditingEnabled) {
assertThat(source.getValue("audit.outputs"), contains(LoggingAuditTrail.NAME));
} else {
assertThat(source.getValue("audit.outputs"), is(nullValue()));
}
// ip filter
assertThat(source.getValue("ipfilter.http.enabled"), is(httpIpFilterEnabled));

51
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecuritySettingsTests.java
View File

@ -1,51 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.security;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.xpack.core.XPackSettings;
import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
import static org.hamcrest.Matchers.containsString;
public class SecuritySettingsTests extends ESTestCase {
public void testValidAutoCreateIndex() {
Security.validateAutoCreateIndex(Settings.EMPTY);
Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", true).build());
Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", false).build());
Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", ".security,.security-6").build());
Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", ".security*").build());
Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", "*s*").build());
Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", ".s*").build());
Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", "foo").build());
Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", ".security_audit_log*").build());
Security.validateAutoCreateIndex(Settings.builder()
.put("action.auto_create_index", ".security,.security-6")
.put(XPackSettings.AUDIT_ENABLED.getKey(), true)
.build());
try {
Security.validateAutoCreateIndex(Settings.builder()
.put("action.auto_create_index", ".security,.security-6")
.put(XPackSettings.AUDIT_ENABLED.getKey(), true)
.put(Security.AUDIT_OUTPUTS_SETTING.getKey(), randomFrom("index", "logfile,index"))
.build());
fail("IllegalArgumentException expected");
} catch (IllegalArgumentException e) {
assertThat(e.getMessage(), containsString(IndexAuditTrailField.INDEX_NAME_PREFIX));
}
Security.validateAutoCreateIndex(Settings.builder()
.put("action.auto_create_index", ".security_audit_log*,.security,.security-6")
.put(XPackSettings.AUDIT_ENABLED.getKey(), true)
.put(Security.AUDIT_OUTPUTS_SETTING.getKey(), randomFrom("index", "logfile,index"))
.build());
}
}

32
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecurityTests.java
View File

@ -43,7 +43,6 @@ import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions;
import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissionsDefinition;
import org.elasticsearch.xpack.core.ssl.SSLService;
import org.elasticsearch.xpack.security.audit.AuditTrailService;
import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail;
import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail;
import org.elasticsearch.xpack.security.authc.Realms;
import org.hamcrest.Matchers;
@ -180,37 +179,6 @@ public class SecurityTests extends ESTestCase {
assertEquals(0, auditTrailService.getAuditTrails().size());
}
public void testIndexAuditTrail() throws Exception {
Settings settings = Settings.builder()
.put(XPackSettings.AUDIT_ENABLED.getKey(), true)
.put(Security.AUDIT_OUTPUTS_SETTING.getKey(), "index").build();
Collection<Object> components = createComponents(settings);
AuditTrailService service = findComponent(AuditTrailService.class, components);
assertNotNull(service);
assertEquals(1, service.getAuditTrails().size());
assertEquals(IndexAuditTrail.NAME, service.getAuditTrails().get(0).name());
}
public void testIndexAndLoggingAuditTrail() throws Exception {
Settings settings = Settings.builder()
.put(XPackSettings.AUDIT_ENABLED.getKey(), true)
.put(Security.AUDIT_OUTPUTS_SETTING.getKey(), "index,logfile").build();
Collection<Object> components = createComponents(settings);
AuditTrailService service = findComponent(AuditTrailService.class, components);
assertNotNull(service);
assertEquals(2, service.getAuditTrails().size());
assertEquals(IndexAuditTrail.NAME, service.getAuditTrails().get(0).name());
assertEquals(LoggingAuditTrail.NAME, service.getAuditTrails().get(1).name());
}
public void testUnknownOutput() {
Settings settings = Settings.builder()
.put(XPackSettings.AUDIT_ENABLED.getKey(), true)
.put(Security.AUDIT_OUTPUTS_SETTING.getKey(), "foo").build();
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> createComponents(settings));
assertEquals("Unknown audit trail output [foo]", e.getMessage());
}
public void testHttpSettingDefaults() throws Exception {
final Settings defaultSettings = Security.additionalSettings(Settings.EMPTY, true, false);
assertThat(SecurityField.NAME4, equalTo(NetworkModule.TRANSPORT_TYPE_SETTING.get(defaultSettings)));

195
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/index/AuditTrailTests.java
View File

@ -1,195 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.security.audit.index;
import org.elasticsearch.action.ActionFuture;
import org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsResponse;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.support.PlainActionFuture;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.Requests;
import org.elasticsearch.client.ResponseException;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.test.SecurityIntegTestCase;
import org.elasticsearch.test.SecuritySettingsSource;
import org.elasticsearch.xpack.core.security.ScrollHelper;
import org.elasticsearch.xpack.core.security.authc.AuthenticationServiceField;
import org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken;
import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
import org.elasticsearch.xpack.security.audit.AuditTrail;
import org.elasticsearch.xpack.security.audit.AuditTrailService;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import static org.elasticsearch.test.SecuritySettingsSourceField.TEST_PASSWORD_SECURE_STRING;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.iterableWithSize;
import static org.hamcrest.Matchers.nullValue;
public class AuditTrailTests extends SecurityIntegTestCase {
private static final String AUTHENTICATE_USER = "http_user";
private static final String EXECUTE_USER = "exec_user";
private static final String ROLE_CAN_RUN_AS = "can_run_as";
private static final String ROLES = ROLE_CAN_RUN_AS + ":\n" + " run_as: [ '" + EXECUTE_USER + "' ]\n";
@Override
protected boolean addMockHttpTransport() {
return false; // enable http
}
@Override
public Settings nodeSettings(int nodeOrdinal) {
return Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
.put("xpack.security.audit.enabled", true)
.put("xpack.security.audit.outputs", "index")
.putList("xpack.security.audit.index.events.include", "access_denied", "authentication_failed", "run_as_denied")
.build();
}
@Override
public String configRoles() {
return ROLES + super.configRoles();
}
@Override
public String configUsers() {
return super.configUsers()
+ AUTHENTICATE_USER + ":" + SecuritySettingsSource.TEST_PASSWORD_HASHED + "\n"
+ EXECUTE_USER + ":xx_no_password_xx\n";
}
@Override
public String configUsersRoles() {
return super.configUsersRoles()
+ ROLE_CAN_RUN_AS + ":" + AUTHENTICATE_USER + "\n"
+ "monitoring_user:" + EXECUTE_USER;
}
@Override
public boolean transportSSLEnabled() {
return true;
}
public void testAuditAccessDeniedWithRunAsUser() throws Exception {
try {
Request request = new Request("GET", "/.security/_search");
RequestOptions.Builder options = request.getOptions().toBuilder();
options.addHeader("Authorization", UsernamePasswordToken.basicAuthHeaderValue(AUTHENTICATE_USER, TEST_PASSWORD_SECURE_STRING));
options.addHeader(AuthenticationServiceField.RUN_AS_USER_HEADER, EXECUTE_USER);
request.setOptions(options);
getRestClient().performRequest(request);
fail("request should have failed");
} catch (final ResponseException e) {
assertThat(e.getResponse().getStatusLine().getStatusCode(), is(403));
}
final Collection<Map<String, Object>> events = waitForAuditEvents();
assertThat(events, iterableWithSize(1));
final Map<String, Object> event = events.iterator().next();
assertThat(event.get(IndexAuditTrail.Field.TYPE), equalTo("access_denied"));
assertThat((List<?>) event.get(IndexAuditTrail.Field.INDICES), containsInAnyOrder(".security"));
assertThat(event.get(IndexAuditTrail.Field.PRINCIPAL), equalTo(EXECUTE_USER));
assertThat(event.get(IndexAuditTrail.Field.RUN_BY_PRINCIPAL), equalTo(AUTHENTICATE_USER));
}
public void testAuditRunAsDeniedEmptyUser() throws Exception {
try {
Request request = new Request("GET", "/.security/_search");
RequestOptions.Builder options = request.getOptions().toBuilder();
options.addHeader("Authorization", UsernamePasswordToken.basicAuthHeaderValue(AUTHENTICATE_USER, TEST_PASSWORD_SECURE_STRING));
options.addHeader(AuthenticationServiceField.RUN_AS_USER_HEADER, "");
request.setOptions(options);
getRestClient().performRequest(request);
fail("request should have failed");
} catch (final ResponseException e) {
assertThat(e.getResponse().getStatusLine().getStatusCode(), is(401));
}
final Collection<Map<String, Object>> events = waitForAuditEvents();
assertThat(events, iterableWithSize(1));
final Map<String, Object> event = events.iterator().next();
assertThat(event.get(IndexAuditTrail.Field.TYPE), equalTo("run_as_denied"));
assertThat(event.get(IndexAuditTrail.Field.PRINCIPAL), equalTo(AUTHENTICATE_USER));
assertThat(event.get(IndexAuditTrail.Field.RUN_AS_PRINCIPAL), equalTo(""));
assertThat(event.get(IndexAuditTrail.Field.REALM), equalTo("file"));
assertThat(event.get(IndexAuditTrail.Field.RUN_AS_REALM), nullValue());
}
private Collection<Map<String, Object>> waitForAuditEvents() throws InterruptedException {
waitForAuditTrailToBeWritten();
final AtomicReference<Collection<Map<String, Object>>> eventsRef = new AtomicReference<>();
awaitBusy(() -> {
try {
final Collection<Map<String, Object>> events = getAuditEvents();
eventsRef.set(events);
return events.size() > 0;
} catch (final Exception e) {
throw new RuntimeException(e);
}
});
return eventsRef.get();
}
private Collection<Map<String, Object>> getAuditEvents() throws Exception {
final Client client = client();
final DateTime now = new DateTime(DateTimeZone.UTC);
final String indexName = IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now, IndexNameResolver.Rollover.DAILY);
assertTrue(awaitBusy(() -> indexExists(client, indexName), 5, TimeUnit.SECONDS));
client.admin().indices().refresh(Requests.refreshRequest(indexName)).get();
final SearchRequest request = client.prepareSearch(indexName)
.setScroll(TimeValue.timeValueMinutes(10L))
.setTypes(IndexAuditTrail.DOC_TYPE)
.setQuery(QueryBuilders.matchAllQuery())
.setSize(1000)
.setFetchSource(true)
.request();
request.indicesOptions().ignoreUnavailable();
final PlainActionFuture<Collection<Map<String, Object>>> listener = new PlainActionFuture<>();
ScrollHelper.fetchAllByEntity(client, request, listener, SearchHit::getSourceAsMap);
return listener.get();
}
private boolean indexExists(Client client, String indexName) {
try {
final ActionFuture<IndicesExistsResponse> future = client.admin().indices().exists(Requests.indicesExistsRequest(indexName));
return future.get().isExists();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException("Failed to check if " + indexName + " exists", e);
}
}
private void waitForAuditTrailToBeWritten() throws InterruptedException {
final AuditTrailService auditTrailService = (AuditTrailService) internalCluster().getInstance(AuditTrail.class);
assertThat(auditTrailService.getAuditTrails(), iterableWithSize(1));
final IndexAuditTrail indexAuditTrail = (IndexAuditTrail) auditTrailService.getAuditTrails().get(0);
assertTrue(awaitBusy(() -> indexAuditTrail.peek() == null, 5, TimeUnit.SECONDS));
}
}

324
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/index/IndexAuditTrailMutedTests.java
View File

@ -1,324 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.security.audit.index;
import org.elasticsearch.action.Action;
import org.elasticsearch.action.ActionListener;
import org.elasticsearch.action.ActionRequest;
import org.elasticsearch.action.ActionResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.FilterClient;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.cluster.service.ClusterService;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.rest.RestRequest;
import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.threadpool.TestThreadPool;
import org.elasticsearch.threadpool.ThreadPool;
import org.elasticsearch.transport.MockTransportClient;
import org.elasticsearch.transport.TransportMessage;
import org.elasticsearch.xpack.core.security.authc.Authentication;
import org.elasticsearch.xpack.core.security.authc.Authentication.RealmRef;
import org.elasticsearch.xpack.core.security.authc.AuthenticationToken;
import org.elasticsearch.xpack.core.security.user.SystemUser;
import org.elasticsearch.xpack.core.security.user.User;
import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.State;
import org.elasticsearch.xpack.security.transport.filter.SecurityIpFilterRule;
import org.junit.After;
import org.junit.Before;
import java.net.InetAddress;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.atomic.AtomicBoolean;
import static org.hamcrest.Matchers.is;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verifyZeroInteractions;
import static org.mockito.Mockito.when;
public class IndexAuditTrailMutedTests extends ESTestCase {
private Client client;
private TransportClient transportClient;
private ThreadPool threadPool;
private ClusterService clusterService;
private IndexAuditTrail auditTrail;
private AtomicBoolean messageEnqueued;
private AtomicBoolean clientCalled;
@Before
public void setup() {
DiscoveryNode localNode = mock(DiscoveryNode.class);
when(localNode.getHostAddress()).thenReturn(buildNewFakeTransportAddress().toString());
clusterService = mock(ClusterService.class);
when(clusterService.localNode()).thenReturn(localNode);
threadPool = new TestThreadPool("index audit trail tests");
transportClient = new MockTransportClient(Settings.EMPTY);
clientCalled = new AtomicBoolean(false);
class IClient extends FilterClient {
IClient(Client transportClient){
super(Settings.EMPTY, threadPool, transportClient);
}
@Override
protected <Request extends ActionRequest, Response extends ActionResponse>
void doExecute(Action<Response> action, Request request, ActionListener<Response> listener) {
clientCalled.set(true);
}
}
client = new IClient(transportClient);
messageEnqueued = new AtomicBoolean(false);
}
@After
public void stop() {
if (auditTrail != null) {
auditTrail.stop();
}
if (transportClient != null) {
transportClient.close();
}
threadPool.shutdown();
}
public void testAnonymousAccessDeniedMutedTransport() {
createAuditTrail(new String[] { "anonymous_access_denied" });
TransportMessage message = mock(TransportMessage.class);
auditTrail.anonymousAccessDenied(randomAlphaOfLengthBetween(6, 12), "_action", message);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(message);
}
public void testAnonymousAccessDeniedMutedRest() {
createAuditTrail(new String[] { "anonymous_access_denied" });
RestRequest restRequest = mock(RestRequest.class);
auditTrail.anonymousAccessDenied(randomAlphaOfLengthBetween(6, 12), restRequest);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(restRequest);
}
public void testAuthenticationFailedMutedTransport() {
createAuditTrail(new String[] { "authentication_failed" });
TransportMessage message = mock(TransportMessage.class);
AuthenticationToken token = mock(AuthenticationToken.class);
// without realm
auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), token, "_action", message);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
// without the token
auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), "_action", message);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(token, message);
}
public void testAuthenticationFailedMutedRest() {
createAuditTrail(new String[] { "authentication_failed" });
RestRequest restRequest = mock(RestRequest.class);
AuthenticationToken token = mock(AuthenticationToken.class);
// without the realm
auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), token, restRequest);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
// without the token
auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), restRequest);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(token, restRequest);
}
public void testAuthenticationFailedRealmMutedTransport() {
createAuditTrail(new String[] { "realm_authentication_failed" });
TransportMessage message = mock(TransportMessage.class);
AuthenticationToken token = mock(AuthenticationToken.class);
// with realm
auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), randomAlphaOfLengthBetween(2, 10), token, "_action", message);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(token, message);
}
public void testAuthenticationFailedRealmMutedRest() {
createAuditTrail(new String[]{"realm_authentication_failed"});
RestRequest restRequest = mock(RestRequest.class);
AuthenticationToken token = mock(AuthenticationToken.class);
// with realm
auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), randomAlphaOfLengthBetween(2, 10), token, restRequest);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(token, restRequest);
}
public void testAccessGrantedMuted() {
createAuditTrail(new String[] { "access_granted" });
final TransportMessage message = mock(TransportMessage.class);
final Authentication authentication = mock(Authentication.class);
auditTrail.accessGranted(randomAlphaOfLengthBetween(6, 12), authentication, randomAlphaOfLengthBetween(6, 40), message,
new String[] { "role" });
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(message);
}
public void testSystemAccessGrantedMuted() {
createAuditTrail(randomFrom(new String[] { "access_granted" }, null));
final TransportMessage message = mock(TransportMessage.class);
final Authentication authentication = new Authentication(SystemUser.INSTANCE, new RealmRef(null, null, null), null);
auditTrail.accessGranted(randomAlphaOfLengthBetween(6, 12), authentication, "internal:foo", message, new String[] { "role" });
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(message);
}
public void testAccessDeniedMuted() {
createAuditTrail(new String[] { "access_denied" });
final TransportMessage message = mock(TransportMessage.class);
final Authentication authentication = mock(Authentication.class);
auditTrail.accessDenied(randomAlphaOfLengthBetween(6, 12), authentication, randomAlphaOfLengthBetween(6, 40), message,
new String[] { "role" });
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(message, authentication);
}
public void testTamperedRequestMuted() {
createAuditTrail(new String[] { "tampered_request" });
TransportMessage message = mock(TransportMessage.class);
User user = mock(User.class);
// with user
auditTrail.tamperedRequest(randomAlphaOfLengthBetween(6, 12), user, randomAlphaOfLengthBetween(6, 40), message);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
// without user
auditTrail.tamperedRequest(randomAlphaOfLengthBetween(6, 12), randomAlphaOfLengthBetween(6, 40), message);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(message, user);
}
public void testConnectionGrantedMuted() {
createAuditTrail(new String[] { "connection_granted" });
InetAddress address = mock(InetAddress.class);
SecurityIpFilterRule rule = mock(SecurityIpFilterRule.class);
auditTrail.connectionGranted(address, randomAlphaOfLengthBetween(1, 12), rule);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(address, rule);
}
public void testConnectionDeniedMuted() {
createAuditTrail(new String[] { "connection_denied" });
InetAddress address = mock(InetAddress.class);
SecurityIpFilterRule rule = mock(SecurityIpFilterRule.class);
auditTrail.connectionDenied(address, randomAlphaOfLengthBetween(1, 12), rule);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(address, rule);
}
public void testRunAsGrantedMuted() {
createAuditTrail(new String[] { "run_as_granted" });
TransportMessage message = mock(TransportMessage.class);
Authentication authentication = mock(Authentication.class);
auditTrail.runAsGranted(randomAlphaOfLengthBetween(6, 12), authentication, randomAlphaOfLengthBetween(6, 40), message,
new String[] { "role" });
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(message, authentication);
}
public void testRunAsDeniedMuted() {
createAuditTrail(new String[] { "run_as_denied" });
TransportMessage message = mock(TransportMessage.class);
Authentication authentication = mock(Authentication.class);
auditTrail.runAsDenied(randomAlphaOfLengthBetween(6, 12), authentication, randomAlphaOfLengthBetween(6, 40), message,
new String[] { "role" });
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(message, authentication);
}
public void testAuthenticationSuccessRest() {
createAuditTrail(new String[] { "authentication_success" });
RestRequest restRequest = mock(RestRequest.class);
User user = mock(User.class);
String realm = "_realm";
auditTrail.authenticationSuccess(randomAlphaOfLengthBetween(6, 12), realm, user, restRequest);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(restRequest);
}
public void testAuthenticationSuccessTransport() {
createAuditTrail(new String[] { "authentication_success" });
TransportMessage message = mock(TransportMessage.class);
User user = mock(User.class);
String realm = "_realm";
auditTrail.authenticationSuccess(randomAlphaOfLengthBetween(6, 12), realm, user, randomAlphaOfLengthBetween(6, 40), message);
assertThat(messageEnqueued.get(), is(false));
assertThat(clientCalled.get(), is(false));
verifyZeroInteractions(message, user);
}
IndexAuditTrail createAuditTrail(String[] excludes) {
Settings settings = IndexAuditTrailTests.levelSettings(null, excludes);
auditTrail = new IndexAuditTrail(settings, client, threadPool, clusterService) {
@Override
void updateCurrentIndexMappingsIfNecessary(ClusterState state) {
// skip stuff so we don't have to stub out unnecessary client activities and cluster state
innerStart();
}
@Override
BlockingQueue<Message> createQueue(int maxQueueSize) {
return new LinkedBlockingQueue<Message>(maxQueueSize) {
@Override
public boolean offer(Message message) {
messageEnqueued.set(true);
return super.offer(message);
}
};
}
};
auditTrail.start();
assertThat(auditTrail.state(), is(State.STARTED));
return auditTrail;
}
}

996
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/index/IndexAuditTrailTests.java
View File

@ -1,996 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.security.audit.index;
import org.apache.lucene.util.SetOnce;
import org.elasticsearch.action.IndicesRequest;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
import org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse;
import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesRequest;
import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.support.IndicesOptions;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.Requests;
import org.elasticsearch.client.transport.NoNodeAvailableException;
import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.health.ClusterHealthStatus;
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.cluster.node.DiscoveryNodes;
import org.elasticsearch.cluster.service.ClusterService;
import org.elasticsearch.common.Priority;
import org.elasticsearch.common.network.NetworkAddress;
import org.elasticsearch.common.network.NetworkModule;
import org.elasticsearch.common.settings.KeyStoreWrapper;
import org.elasticsearch.common.settings.MockSecureSettings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.common.util.concurrent.EsExecutors;
import org.elasticsearch.common.util.set.Sets;
import org.elasticsearch.discovery.DiscoveryModule;
import org.elasticsearch.discovery.zen.SettingsBasedHostsProvider;
import org.elasticsearch.http.HttpChannel;
import org.elasticsearch.plugins.MetaDataUpgrader;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.rest.RestRequest;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.InternalTestCluster;
import org.elasticsearch.test.SecurityIntegTestCase;
import org.elasticsearch.test.SecuritySettingsSource;
import org.elasticsearch.test.SecuritySettingsSourceField;
import org.elasticsearch.test.discovery.TestZenDiscovery;
import org.elasticsearch.threadpool.TestThreadPool;
import org.elasticsearch.threadpool.ThreadPool;
import org.elasticsearch.transport.TransportInfo;
import org.elasticsearch.transport.TransportMessage;
import org.elasticsearch.transport.TransportRequest;
import org.elasticsearch.xpack.core.XPackSettings;
import org.elasticsearch.xpack.core.security.SecurityField;
import org.elasticsearch.xpack.core.security.authc.Authentication;
import org.elasticsearch.xpack.core.security.authc.Authentication.RealmRef;
import org.elasticsearch.xpack.core.security.authc.AuthenticationToken;
import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
import org.elasticsearch.xpack.core.security.user.SystemUser;
import org.elasticsearch.xpack.core.security.user.User;
import org.elasticsearch.xpack.security.LocalStateSecurity;
import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.Message;
import org.elasticsearch.xpack.security.support.SecurityIndexManager;
import org.elasticsearch.xpack.security.transport.filter.IPFilter;
import org.elasticsearch.xpack.security.transport.filter.SecurityIpFilterRule;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.format.ISODateTimeFormat;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.function.Function;
import static java.util.Collections.emptyMap;
import static org.elasticsearch.test.ESIntegTestCase.Scope.SUITE;
import static org.elasticsearch.test.InternalTestCluster.clusterName;
import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.Rollover.DAILY;
import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.Rollover.HOURLY;
import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.Rollover.MONTHLY;
import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.Rollover.WEEKLY;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.hasToString;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.notNullValue;
import static org.hamcrest.Matchers.nullValue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
@ESIntegTestCase.ClusterScope(scope = SUITE, supportsDedicatedMasters = false, numDataNodes = 1)
public class IndexAuditTrailTests extends SecurityIntegTestCase {
public static final String SECOND_CLUSTER_NODE_PREFIX = "remote_" + SUITE_CLUSTER_NODE_PREFIX;
private static boolean remoteIndexing;
private static boolean useSSL;
private static InternalTestCluster remoteCluster;
private static Settings remoteSettings;
private static int numShards = -1;
private static int numReplicas = -1;
private TransportAddress remoteAddress = buildNewFakeTransportAddress();
private TransportAddress localAddress = new TransportAddress(InetAddress.getLoopbackAddress(), 0);
private IndexNameResolver.Rollover rollover;
private IndexAuditTrail auditor;
private SetOnce<Message> enqueuedMessage;
private ThreadPool threadPool;
private boolean includeRequestBody;
@BeforeClass
public static void configureBeforeClass() {
useSSL = randomBoolean();
remoteIndexing = randomBoolean();
if (remoteIndexing == false) {
remoteSettings = Settings.EMPTY;
}
}
@AfterClass
public static void cleanupAfterTest() {
if (remoteCluster != null) {
remoteCluster.close();
remoteCluster = null;
}
remoteSettings = null;
}
@Override
protected boolean transportSSLEnabled() {
return useSSL;
}
@Override
public Settings nodeSettings(int nodeOrdinal) {
if (numShards == -1) {
numShards = numberOfShards();
}
if (numReplicas == -1) {
numReplicas = numberOfReplicas();
}
return Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
.put("xpack.security.audit.index.settings.index.number_of_shards", numShards)
.put("xpack.security.audit.index.settings.index.number_of_replicas", numReplicas)
.build();
}
@Before
public void initializeRemoteClusterIfNecessary() throws Exception {
if (remoteIndexing == false) {
logger.info("--> remote indexing disabled.");
return;
}
if (remoteCluster != null) {
return;
}
// create another cluster
String cluster2Name = clusterName(Scope.SUITE.name(), randomLong());
// Setup a second test cluster with randomization for number of nodes, security enabled, and SSL
final int numNodes = randomIntBetween(1, 2);
final boolean useSecurity = randomBoolean();
final boolean remoteUseSSL = useSecurity && useSSL;
logger.info("--> remote indexing enabled. security enabled: [{}], SSL enabled: [{}], nodes: [{}]", useSecurity, useSSL,
numNodes);
SecuritySettingsSource cluster2SettingsSource =
new SecuritySettingsSource(useSSL, createTempDir(), Scope.SUITE) {
@Override
public Settings nodeSettings(int nodeOrdinal) {
Settings.Builder builder = Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
.put(DiscoveryModule.DISCOVERY_HOSTS_PROVIDER_SETTING.getKey(), "file")
.putList(SettingsBasedHostsProvider.DISCOVERY_ZEN_PING_UNICAST_HOSTS_SETTING.getKey())
.put(TestZenDiscovery.USE_ZEN2.getKey(), getUseZen2())
.put("xpack.security.audit.index.settings.index.number_of_shards", numShards)
.put("xpack.security.audit.index.settings.index.number_of_replicas", numReplicas)
// Disable native ML autodetect_process as the c++ controller won't be available
// .put(MachineLearningField.AUTODETECT_PROCESS.getKey(), false)
.put(XPackSettings.SECURITY_ENABLED.getKey(), useSecurity);
String transport = builder.get(NetworkModule.TRANSPORT_TYPE_KEY);
if (useSecurity == false && (transport == null || SecurityField.NAME4.equals(transport)
|| SecurityField.NIO.equals(transport))) {
builder.put(NetworkModule.TRANSPORT_TYPE_KEY, getTestTransportType());
}
return builder.build();
}
@Override
public Settings transportClientSettings() {
if (useSecurity) {
return super.transportClientSettings();
} else {
Settings.Builder builder = Settings.builder()
.put(XPackSettings.SECURITY_ENABLED.getKey(), false)
.put(super.transportClientSettings());
if (builder.get(NetworkModule.TRANSPORT_TYPE_KEY) == null) {
builder.put(NetworkModule.TRANSPORT_TYPE_KEY, getTestTransportType());
}
return builder.build();
}
}
@Override
protected void addDefaultSecurityTransportType(Settings.Builder builder, Settings settings) {
if (useSecurity) {
super.addDefaultSecurityTransportType(builder, settings);
}
}
};
Set<Class<? extends Plugin>> mockPlugins = new HashSet<>(getMockPlugins());
if (useSecurity == false) {
mockPlugins.add(getTestTransportPlugin());
}
remoteCluster = new InternalTestCluster(randomLong(), createTempDir(), false, true, numNodes, numNodes, cluster2Name,
cluster2SettingsSource, 0, SECOND_CLUSTER_NODE_PREFIX, mockPlugins,
useSecurity ? getClientWrapper() : Function.identity());
remoteCluster.beforeTest(random(), 0.5);
NodesInfoResponse response = remoteCluster.client().admin().cluster().prepareNodesInfo().execute().actionGet();
TransportInfo info = response.getNodes().get(0).getTransport();
TransportAddress inet = info.address().publishAddress();
Settings.Builder builder = Settings.builder()
.put("xpack.security.audit.index.client." + XPackSettings.SECURITY_ENABLED.getKey(), useSecurity)
.put(remoteSettings(NetworkAddress.format(inet.address().getAddress()), inet.address().getPort(), cluster2Name))
.put("xpack.security.audit.index.client.xpack.security.user", SecuritySettingsSource.TEST_USER_NAME + ":" +
SecuritySettingsSourceField.TEST_PASSWORD);
if (remoteUseSSL) {
cluster2SettingsSource.addClientSSLSettings(builder, "xpack.security.audit.index.client.xpack.security.transport.");
builder.put("xpack.security.audit.index.client.xpack.security.transport.ssl.enabled", true);
}
if (useSecurity == false && builder.get(NetworkModule.TRANSPORT_TYPE_KEY) == null) {
builder.put("xpack.security.audit.index.client." + NetworkModule.TRANSPORT_TYPE_KEY, getTestTransportType());
}
remoteSettings = builder.build();
}
@After
public void afterTest() {
if (threadPool != null) {
threadPool.shutdown();
}
if (auditor != null) {
auditor.stop();
}
if (remoteCluster != null) {
remoteCluster.wipe(excludeTemplates());
}
}
@Override
protected Set<String> excludeTemplates() {
return Sets.newHashSet(SecurityIndexManager.SECURITY_TEMPLATE_NAME, IndexAuditTrail.INDEX_TEMPLATE_NAME);
}
@Override
protected int maximumNumberOfShards() {
return 3;
}
private Settings commonSettings(IndexNameResolver.Rollover rollover) {
return Settings.builder()
.put("xpack.security.audit.enabled", true)
.put("xpack.security.audit.outputs", "index, logfile")
.put("xpack.security.audit.index.bulk_size", 1)
.put("xpack.security.audit.index.flush_interval", "1ms")
.put("xpack.security.audit.index.rollover", rollover.name().toLowerCase(Locale.ENGLISH))
.put("xpack.security.audit.index.settings.index.number_of_shards", numShards)
.put("xpack.security.audit.index.settings.index.number_of_replicas", numReplicas)
.build();
}
static Settings remoteSettings(String address, int port, String clusterName) {
return Settings.builder()
.put("xpack.security.audit.index.client.hosts", address + ":" + port)
.put("xpack.security.audit.index.client.cluster.name", clusterName)
.build();
}
static Settings levelSettings(String[] includes, String[] excludes) {
Settings.Builder builder = Settings.builder();
if (includes != null) {
builder.putList("xpack.security.audit.index.events.include", includes);
}
if (excludes != null) {
builder.putList("xpack.security.audit.index.events.exclude", excludes);
}
return builder.build();
}
private Settings settings(IndexNameResolver.Rollover rollover, String[] includes, String[] excludes) {
Settings.Builder builder = Settings.builder();
builder.put(levelSettings(includes, excludes));
builder.put(commonSettings(rollover));
builder.put("xpack.security.audit.index.events.emit_request_body", includeRequestBody);
return builder.build();
}
private Client getClient() {
return remoteIndexing ? remoteCluster.client() : client();
}
private void initialize() throws Exception {
initialize(null, null);
}
private void initialize(String[] includes, String[] excludes) throws Exception {
initialize(includes, excludes, Settings.EMPTY);
}
private void initialize(final String[] includes, final String[] excludes, final Settings additionalSettings) throws Exception {
rollover = randomFrom(HOURLY, DAILY, WEEKLY, MONTHLY);
includeRequestBody = randomBoolean();
Settings.Builder builder = Settings.builder();
if (remoteIndexing) {
builder.put(remoteSettings);
}
builder.put(settings(rollover, includes, excludes)).put(additionalSettings).build();
// IndexAuditTrail should ignore secure settings
// they are merged on the master node creating the audit index
if (randomBoolean()) {
MockSecureSettings ignored = new MockSecureSettings();
if (randomBoolean()) {
ignored.setString(KeyStoreWrapper.SEED_SETTING.getKey(), "non-empty-secure-settings");
}
builder.setSecureSettings(ignored);
}
Settings settings = builder.build();
logger.info("--> settings: [{}]", settings);
DiscoveryNode localNode = mock(DiscoveryNode.class);
when(localNode.getHostAddress()).thenReturn(remoteAddress.getAddress());
when(localNode.getHostName()).thenReturn(remoteAddress.getAddress());
ClusterService clusterService = mock(ClusterService.class);
ClusterState state = mock(ClusterState.class);
DiscoveryNodes nodes = mock(DiscoveryNodes.class);
when(clusterService.localNode()).thenReturn(localNode);
when(clusterService.state()).thenReturn(client().admin().cluster().prepareState().get().getState());
when(state.getNodes()).thenReturn(nodes);
when(nodes.isLocalNodeElectedMaster()).thenReturn(true);
threadPool = new TestThreadPool("index audit trail tests");
enqueuedMessage = new SetOnce<>();
auditor = new IndexAuditTrail(settings, client(), threadPool, clusterService) {
@Override
void enqueue(Message message, String type) {
enqueuedMessage.set(message);
super.enqueue(message, type);
}
@Override
List<Class<? extends Plugin>> remoteTransportClientPlugins() {
return Arrays.asList(LocalStateSecurity.class, getTestTransportPlugin());
}
};
auditor.start();
}
public void testIndexTemplateUpgrader() throws Exception {
final MetaDataUpgrader metaDataUpgrader = internalCluster().getInstance(MetaDataUpgrader.class);
final Map<String, IndexTemplateMetaData> updatedTemplates = metaDataUpgrader.indexTemplateMetaDataUpgraders.apply(emptyMap());
final IndexTemplateMetaData indexAuditTrailTemplate = updatedTemplates.get(IndexAuditTrail.INDEX_TEMPLATE_NAME);
assertThat(indexAuditTrailTemplate, notNullValue());
// test custom index settings override template
assertThat(IndexMetaData.INDEX_NUMBER_OF_REPLICAS_SETTING.get(indexAuditTrailTemplate.settings()), is(numReplicas));
assertThat(IndexMetaData.INDEX_NUMBER_OF_SHARDS_SETTING.get(indexAuditTrailTemplate.settings()), is(numShards));
// test upgrade template and installed template are equal
final GetIndexTemplatesRequest request = new GetIndexTemplatesRequest(IndexAuditTrail.INDEX_TEMPLATE_NAME);
final GetIndexTemplatesResponse response = client().admin().indices().getTemplates(request).get();
assertThat(response.getIndexTemplates(), hasSize(1));
assertThat(indexAuditTrailTemplate, is(response.getIndexTemplates().get(0)));
}
public void testProcessorsSetting() {
final boolean explicitProcessors = randomBoolean();
final int processors;
if (explicitProcessors) {
processors = randomIntBetween(1, 16);
} else {
processors = EsExecutors.PROCESSORS_SETTING.get(Settings.EMPTY);
}
final boolean explicitClientProcessors = randomBoolean();
final int clientProcessors;
if (explicitClientProcessors) {
clientProcessors = randomIntBetween(1, 16);
} else {
clientProcessors = EsExecutors.PROCESSORS_SETTING.get(Settings.EMPTY);
}
final Settings.Builder additionalSettingsBuilder =
Settings.builder()
.put("xpack.security.audit.index.client.cluster.name", "remote")
.put("xpack.security.audit.index.client.hosts", "localhost:9300");
if (explicitProcessors) {
additionalSettingsBuilder.put(EsExecutors.PROCESSORS_SETTING.getKey(), processors);
}
if (explicitClientProcessors) {
additionalSettingsBuilder.put("xpack.security.audit.index.client.processors", clientProcessors);
}
final ThrowingRunnable runnable = () -> initialize(null, null, additionalSettingsBuilder.build());
if (processors == clientProcessors || explicitClientProcessors == false) {
// okay, the client initialized which is all we care about but no nodes are available because we never set up the remote cluster
expectThrows(NoNodeAvailableException.class, runnable);
} else {
final IllegalStateException e = expectThrows(IllegalStateException.class, runnable);
assertThat(
e,
hasToString(containsString(
"explicit processor setting [" + clientProcessors + "]" +
" for audit trail remote client does not match inherited processor setting [" + processors + "]")));
}
}
public void testAnonymousAccessDeniedTransport() throws Exception {
initialize();
TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
auditor.anonymousAccessDenied(randomAlphaOfLengthBetween(6, 12), "_action", message);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "transport", "anonymous_access_denied");
Map<String, Object> sourceMap = hit.getSourceAsMap();
if (message instanceof RemoteHostMockMessage) {
assertEquals(remoteAddress.getAddress(), sourceMap.get("origin_address"));
} else {
assertEquals(localAddress.getAddress(), sourceMap.get("origin_address"));
}
assertEquals("_action", sourceMap.get("action"));
assertEquals("transport", sourceMap.get("origin_type"));
if (message instanceof IndicesRequest) {
List<Object> indices = (List<Object>) sourceMap.get("indices");
assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
}
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testAnonymousAccessDeniedRest() throws Exception {
initialize();
RestRequest request = mockRestRequest();
auditor.anonymousAccessDenied(randomAlphaOfLengthBetween(6, 12), request);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "rest", "anonymous_access_denied");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertThat(NetworkAddress.format(InetAddress.getLoopbackAddress()), equalTo(sourceMap.get("origin_address")));
assertThat("_uri", equalTo(sourceMap.get("uri")));
assertThat(sourceMap.get("origin_type"), is("rest"));
assertRequestBody(sourceMap);
}
public void testAuthenticationFailedTransport() throws Exception {
initialize();
TransportMessage message = randomBoolean() ? new RemoteHostMockMessage() : new LocalHostMockMessage();
auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), new MockToken(), "_action", message);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertAuditMessage(hit, "transport", "authentication_failed");
if (message instanceof RemoteHostMockMessage) {
assertEquals(remoteAddress.getAddress(), sourceMap.get("origin_address"));
} else {
assertEquals(localAddress.getAddress(), sourceMap.get("origin_address"));
}
assertEquals("_principal", sourceMap.get("principal"));
assertEquals("_action", sourceMap.get("action"));
assertEquals("transport", sourceMap.get("origin_type"));
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testAuthenticationFailedTransportNoToken() throws Exception {
initialize();
TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), "_action", message);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "transport", "authentication_failed");
Map<String, Object> sourceMap = hit.getSourceAsMap();
if (message instanceof RemoteHostMockMessage) {
assertEquals(remoteAddress.getAddress(), sourceMap.get("origin_address"));
} else {
assertEquals(localAddress.getAddress(), sourceMap.get("origin_address"));
}
assertThat(sourceMap.get("principal"), nullValue());
assertEquals("_action", sourceMap.get("action"));
assertEquals("transport", sourceMap.get("origin_type"));
if (message instanceof IndicesRequest) {
List<Object> indices = (List<Object>) sourceMap.get("indices");
assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
}
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testAuthenticationFailedRest() throws Exception {
initialize();
RestRequest request = mockRestRequest();
auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), new MockToken(), request);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "rest", "authentication_failed");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertThat(sourceMap.get("principal"), is((Object) "_principal"));
assertThat("127.0.0.1", equalTo(sourceMap.get("origin_address")));
assertThat("_uri", equalTo(sourceMap.get("uri")));
assertThat(sourceMap.get("origin_type"), is("rest"));
assertRequestBody(sourceMap);
}
public void testAuthenticationFailedRestNoToken() throws Exception {
initialize();
RestRequest request = mockRestRequest();
auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), request);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "rest", "authentication_failed");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertThat(sourceMap.get("principal"), nullValue());
assertThat("127.0.0.1", equalTo(sourceMap.get("origin_address")));
assertThat("_uri", equalTo(sourceMap.get("uri")));
assertThat(sourceMap.get("origin_type"), is("rest"));
assertRequestBody(sourceMap);
}
public void testAuthenticationFailedTransportRealm() throws Exception {
initialize();
TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), "_realm", new MockToken(), "_action", message);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "transport", "realm_authentication_failed");
Map<String, Object> sourceMap = hit.getSourceAsMap();
if (message instanceof RemoteHostMockMessage) {
assertEquals(remoteAddress.getAddress(), sourceMap.get("origin_address"));
} else {
assertEquals(localAddress.getAddress(), sourceMap.get("origin_address"));
}
assertEquals("transport", sourceMap.get("origin_type"));
assertEquals("_principal", sourceMap.get("principal"));
assertEquals("_action", sourceMap.get("action"));
assertEquals("_realm", sourceMap.get("realm"));
if (message instanceof IndicesRequest) {
List<Object> indices = (List<Object>) sourceMap.get("indices");
assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
}
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testAuthenticationFailedRestRealm() throws Exception {
initialize();
RestRequest request = mockRestRequest();
auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), "_realm", new MockToken(), request);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "rest", "realm_authentication_failed");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertThat("127.0.0.1", equalTo(sourceMap.get("origin_address")));
assertThat("_uri", equalTo(sourceMap.get("uri")));
assertEquals("_realm", sourceMap.get("realm"));
assertThat(sourceMap.get("origin_type"), is("rest"));
assertRequestBody(sourceMap);
}
public void testAccessGranted() throws Exception {
initialize();
TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
final boolean runAs = randomBoolean();
User user;
if (runAs) {
user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
} else {
user = new User("_username", new String[]{"r1"});
}
String role = randomAlphaOfLengthBetween(1, 6);
auditor.accessGranted(randomAlphaOfLengthBetween(6, 12), createAuthentication(user), "_action", message, new String[] { role });
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "transport", "access_granted");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertEquals("transport", sourceMap.get("origin_type"));
if (runAs) {
assertThat(sourceMap.get("principal"), is("running as"));
assertThat(sourceMap.get("realm"), is("lookRealm"));
assertThat(sourceMap.get("run_by_principal"), is("_username"));
assertThat(sourceMap.get("run_by_realm"), is("authRealm"));
} else {
assertThat(sourceMap.get("principal"), is("_username"));
assertThat(sourceMap.get("realm"), is("authRealm"));
}
assertEquals("_action", sourceMap.get("action"));
assertThat((Iterable<String>) sourceMap.get(IndexAuditTrail.Field.ROLE_NAMES), containsInAnyOrder(role));
if (message instanceof IndicesRequest) {
List<Object> indices = (List<Object>) sourceMap.get("indices");
assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
}
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testSystemAccessGranted() throws Exception {
initialize(new String[] { "system_access_granted" }, null);
TransportMessage message = randomBoolean() ? new RemoteHostMockMessage() : new LocalHostMockMessage();
String role = randomAlphaOfLengthBetween(1, 6);
auditor.accessGranted(randomAlphaOfLength(8), createAuthentication(SystemUser.INSTANCE), "internal:_action", message,
new String[] { role });
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "transport", "access_granted");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertEquals("transport", sourceMap.get("origin_type"));
assertEquals(SystemUser.INSTANCE.principal(), sourceMap.get("principal"));
assertThat(sourceMap.get("realm"), is("authRealm"));
assertEquals("internal:_action", sourceMap.get("action"));
assertThat((Iterable<String>) sourceMap.get(IndexAuditTrail.Field.ROLE_NAMES), containsInAnyOrder(role));
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testAccessDenied() throws Exception {
initialize();
TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
final boolean runAs = randomBoolean();
User user;
if (runAs) {
user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
} else {
user = new User("_username", new String[]{"r1"});
}
String role = randomAlphaOfLengthBetween(1, 6);
auditor.accessDenied(randomAlphaOfLengthBetween(6, 12), createAuthentication(user), "_action", message, new String[] { role });
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertAuditMessage(hit, "transport", "access_denied");
assertEquals("transport", sourceMap.get("origin_type"));
if (runAs) {
assertThat(sourceMap.get("principal"), is("running as"));
assertThat(sourceMap.get("realm"), is("lookRealm"));
assertThat(sourceMap.get("run_by_principal"), is("_username"));
assertThat(sourceMap.get("run_by_realm"), is("authRealm"));
} else {
assertThat(sourceMap.get("principal"), is("_username"));
assertThat(sourceMap.get("realm"), is("authRealm"));
}
assertEquals("_action", sourceMap.get("action"));
if (message instanceof IndicesRequest) {
List<Object> indices = (List<Object>) sourceMap.get("indices");
assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
}
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
assertThat((Iterable<String>) sourceMap.get(IndexAuditTrail.Field.ROLE_NAMES), containsInAnyOrder(role));
}
public void testTamperedRequestRest() throws Exception {
initialize();
RestRequest request = mockRestRequest();
auditor.tamperedRequest(randomAlphaOfLengthBetween(6, 12), request);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "rest", "tampered_request");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertThat(sourceMap.get("principal"), nullValue());
assertThat("127.0.0.1", equalTo(sourceMap.get("origin_address")));
assertThat("_uri", equalTo(sourceMap.get("uri")));
assertThat(sourceMap.get("origin_type"), is("rest"));
assertRequestBody(sourceMap);
}
public void testTamperedRequest() throws Exception {
initialize();
TransportRequest message = new RemoteHostMockTransportRequest();
auditor.tamperedRequest(randomAlphaOfLengthBetween(6, 12), "_action", message);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertAuditMessage(hit, "transport", "tampered_request");
assertEquals("transport", sourceMap.get("origin_type"));
assertThat(sourceMap.get("principal"), is(nullValue()));
assertEquals("_action", sourceMap.get("action"));
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testTamperedRequestWithUser() throws Exception {
initialize();
TransportRequest message = new RemoteHostMockTransportRequest();
final boolean runAs = randomBoolean();
User user;
if (runAs) {
user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
} else {
user = new User("_username", new String[]{"r1"});
}
auditor.tamperedRequest(randomAlphaOfLengthBetween(6, 12), user, "_action", message);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "transport", "tampered_request");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertEquals("transport", sourceMap.get("origin_type"));
if (runAs) {
assertThat(sourceMap.get("principal"), is("running as"));
assertThat(sourceMap.get("run_by_principal"), is("_username"));
} else {
assertEquals("_username", sourceMap.get("principal"));
}
assertEquals("_action", sourceMap.get("action"));
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testConnectionGranted() throws Exception {
initialize();
InetAddress inetAddress = InetAddress.getLoopbackAddress();
SecurityIpFilterRule rule = IPFilter.DEFAULT_PROFILE_ACCEPT_ALL;
auditor.connectionGranted(inetAddress, "default", rule);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "ip_filter", "connection_granted");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertEquals("allow default:accept_all", sourceMap.get("rule"));
assertEquals("default", sourceMap.get("transport_profile"));
}
public void testConnectionDenied() throws Exception {
initialize();
InetAddress inetAddress = InetAddress.getLoopbackAddress();
SecurityIpFilterRule rule = new SecurityIpFilterRule(false, "_all");
auditor.connectionDenied(inetAddress, "default", rule);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "ip_filter", "connection_denied");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertEquals("deny _all", sourceMap.get("rule"));
assertEquals("default", sourceMap.get("transport_profile"));
}
public void testRunAsGranted() throws Exception {
initialize();
TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
User user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
String role = randomAlphaOfLengthBetween(1, 6);
auditor.runAsGranted(randomAlphaOfLengthBetween(6, 12), createAuthentication(user), "_action", message, new String[] { role });
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "transport", "run_as_granted");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertEquals("transport", sourceMap.get("origin_type"));
assertThat(sourceMap.get("principal"), is("_username"));
assertThat(sourceMap.get("realm"), is("authRealm"));
assertThat(sourceMap.get("run_as_principal"), is("running as"));
assertThat(sourceMap.get("run_as_realm"), is("lookRealm"));
assertThat((Iterable<String>) sourceMap.get(IndexAuditTrail.Field.ROLE_NAMES), containsInAnyOrder(role));
assertEquals("_action", sourceMap.get("action"));
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testRunAsDenied() throws Exception {
initialize();
TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
User user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
auditor.runAsDenied(randomAlphaOfLengthBetween(6, 12), createAuthentication(user), "_action", message, new String[] { "r1" });
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "transport", "run_as_denied");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertEquals("transport", sourceMap.get("origin_type"));
assertThat(sourceMap.get("principal"), is("_username"));
assertThat(sourceMap.get("realm"), is("authRealm"));
assertThat(sourceMap.get("run_as_principal"), is("running as"));
assertThat(sourceMap.get("run_as_realm"), is("lookRealm"));
assertEquals("_action", sourceMap.get("action"));
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
public void testAuthenticationSuccessRest() throws Exception {
initialize();
RestRequest request = mockRestRequest();
final boolean runAs = randomBoolean();
User user;
if (runAs) {
user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
} else {
user = new User("_username", new String[] { "r1" });
}
String realm = "_realm";
auditor.authenticationSuccess(randomAlphaOfLengthBetween(6, 12), realm, user, request);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
assertAuditMessage(hit, "rest", "authentication_success");
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertThat("_uri", equalTo(sourceMap.get("uri")));
assertRequestBody(sourceMap);
if (runAs) {
assertThat(sourceMap.get("principal"), is("running as"));
assertThat(sourceMap.get("run_by_principal"), is("_username"));
} else {
assertEquals("_username", sourceMap.get("principal"));
}
assertEquals("_realm", sourceMap.get("realm"));
}
public void testAuthenticationSuccessTransport() throws Exception {
initialize();
TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
final boolean runAs = randomBoolean();
User user;
if (runAs) {
user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
} else {
user = new User("_username", new String[] { "r1" });
}
String realm = "_realm";
auditor.authenticationSuccess(randomAlphaOfLengthBetween(6, 12), realm, user, "_action", message);
SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertAuditMessage(hit, "transport", "authentication_success");
assertEquals("transport", sourceMap.get("origin_type"));
if (runAs) {
assertThat(sourceMap.get("principal"), is("running as"));
assertThat(sourceMap.get("run_by_principal"), is("_username"));
} else {
assertEquals("_username", sourceMap.get("principal"));
}
assertEquals("_action", sourceMap.get("action"));
assertEquals("_realm", sourceMap.get("realm"));
assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
}
private void assertAuditMessage(SearchHit hit, String layer, String type) {
Map<String, Object> sourceMap = hit.getSourceAsMap();
assertThat(sourceMap.get("@timestamp"), notNullValue());
DateTime dateTime = ISODateTimeFormat.dateTimeParser().withZoneUTC().parseDateTime((String) sourceMap.get("@timestamp"));
final DateTime now = DateTime.now(DateTimeZone.UTC);
assertThat(dateTime + " should be on/before " + now, dateTime.isAfter(now), equalTo(false));
assertThat(remoteAddress.getAddress(), equalTo(sourceMap.get("node_host_name")));
assertThat(remoteAddress.getAddress(), equalTo(sourceMap.get("node_host_address")));
assertEquals(layer, sourceMap.get("layer"));
assertEquals(type, sourceMap.get("event_type"));
}
private void assertRequestBody(Map<String, Object> sourceMap) {
if (includeRequestBody) {
assertThat(sourceMap.get("request_body"), notNullValue());
} else {
assertThat(sourceMap.get("request_body"), nullValue());
}
}
private class LocalHostMockMessage extends TransportMessage {
LocalHostMockMessage() {
remoteAddress(localAddress);
}
}
private class RemoteHostMockMessage extends TransportMessage {
RemoteHostMockMessage() throws Exception {
remoteAddress(remoteAddress);
}
}
private class RemoteHostMockTransportRequest extends TransportRequest {
RemoteHostMockTransportRequest() throws Exception {
remoteAddress(remoteAddress);
}
}
private class MockIndicesTransportMessage extends RemoteHostMockMessage implements IndicesRequest {
MockIndicesTransportMessage() throws Exception {
super();
}
@Override
public String[] indices() {
return new String[] { "foo", "bar", "baz" };
}
@Override
public IndicesOptions indicesOptions() {
return null;
}
}
private static class MockToken implements AuthenticationToken {
@Override
public String principal() {
return "_principal";
}
@Override
public Object credentials() {
fail("it's not allowed to print the credentials of the auth token");
return null;
}
@Override
public void clearCredentials() {
}
}
private RestRequest mockRestRequest() {
RestRequest request = mock(RestRequest.class);
HttpChannel httpChannel = mock(HttpChannel.class);
when(request.getHttpChannel()).thenReturn(httpChannel);
when(httpChannel.getRemoteAddress()).thenReturn(new InetSocketAddress(InetAddress.getLoopbackAddress(), 9200));
when(request.uri()).thenReturn("_uri");
return request;
}
private SearchHit getIndexedAuditMessage(Message message) throws InterruptedException {
assertNotNull("no audit message was enqueued", message);
final String indexName = IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, message.timestamp, rollover);
ensureYellowAndNoInitializingShards(indexName);
GetSettingsResponse settingsResponse = getClient().admin().indices().prepareGetSettings(indexName).get();
assertThat(settingsResponse.getSetting(indexName, "index.number_of_shards"), is(Integer.toString(numShards)));
assertThat(settingsResponse.getSetting(indexName, "index.number_of_replicas"), is(Integer.toString(numReplicas)));
final SetOnce<SearchResponse> searchResponseSetOnce = new SetOnce<>();
final boolean found = awaitBusy(() -> {
try {
SearchResponse searchResponse = getClient()
.prepareSearch(indexName)
.setTypes(IndexAuditTrail.DOC_TYPE)
.get();
if (searchResponse.getHits().getTotalHits().value > 0L) {
searchResponseSetOnce.set(searchResponse);
return true;
}
} catch (Exception e) {
logger.debug("caught exception while executing search", e);
}
return false;
});
assertThat("no audit document exists!", found, is(true));
SearchResponse response = searchResponseSetOnce.get();
assertNotNull(response);
assertEquals(1, response.getHits().getTotalHits().value);
return response.getHits().getHits()[0];
}
@Override
public ClusterHealthStatus ensureYellowAndNoInitializingShards(String... indices) {
if (remoteIndexing == false) {
return super.ensureYellowAndNoInitializingShards(indices);
}
// pretty ugly but just a rip of ensureYellowAndNoInitializingShards that uses a different client
ClusterHealthResponse actionGet = getClient().admin().cluster().health(Requests.clusterHealthRequest(indices)
.waitForNoRelocatingShards(true)
.waitForYellowStatus()
.waitForEvents(Priority.LANGUID)
.waitForNoInitializingShards(true))
.actionGet();
if (actionGet.isTimedOut()) {
logger.info("ensureYellow timed out, cluster state:\n{}\n{}",
getClient().admin().cluster().prepareState().get().getState(),
getClient().admin().cluster().preparePendingClusterTasks().get());
assertThat("timed out waiting for yellow", actionGet.isTimedOut(), equalTo(false));
}
logger.debug("indices {} are yellow", indices.length == 0 ? "[_all]" : indices);
return actionGet.getStatus();
}
private static Authentication createAuthentication(User user) {
final RealmRef lookedUpBy = user.authenticatedUser() == user ? null : new RealmRef("lookRealm", "up", "by");
return new Authentication(user, new RealmRef("authRealm", "test", "foo"), lookedUpBy);
}
}

169
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/index/RemoteIndexAuditTrailStartingTests.java
View File

@ -1,169 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.security.audit.index;
import org.elasticsearch.core.internal.io.IOUtils;
import org.elasticsearch.action.admin.cluster.node.info.NodeInfo;
import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.common.util.set.Sets;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import org.elasticsearch.test.ESIntegTestCase.Scope;
import org.elasticsearch.test.InternalTestCluster;
import org.elasticsearch.test.SecurityIntegTestCase;
import org.elasticsearch.test.SecuritySettingsSource;
import org.elasticsearch.test.SecuritySettingsSourceField;
import org.elasticsearch.test.junit.annotations.TestLogging;
import org.elasticsearch.xpack.security.audit.AuditTrail;
import org.elasticsearch.xpack.security.audit.AuditTrailService;
import org.elasticsearch.xpack.security.support.SecurityIndexManager;
import org.junit.After;
import org.junit.Before;
import java.io.Closeable;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.stream.StreamSupport;
import static org.elasticsearch.test.InternalTestCluster.clusterName;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoTimeout;
/**
* This test checks to ensure that the IndexAuditTrail starts properly when indexing to a remote cluster. The cluster
* started by the integration tests is indexed into by the remote cluster started before the test.
*
* The cluster started by the integrations tests may also index into itself...
*/
@ClusterScope(scope = Scope.TEST, numDataNodes = 1, numClientNodes = 0, transportClientRatio = 0.0, supportsDedicatedMasters = false)
@TestLogging("org.elasticsearch.xpack.security.audit.index:TRACE")
public class RemoteIndexAuditTrailStartingTests extends SecurityIntegTestCase {
public static final String SECOND_CLUSTER_NODE_PREFIX = "remote_" + TEST_CLUSTER_NODE_PREFIX;
private InternalTestCluster remoteCluster;
private final boolean sslEnabled = randomBoolean();
private final boolean localAudit = randomBoolean();
private final String outputs = randomFrom("index", "logfile", "index,logfile");
@Override
public boolean transportSSLEnabled() {
return sslEnabled;
}
@Override
public Settings nodeSettings(int nodeOrdinal) {
return Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
.put("xpack.security.audit.enabled", localAudit)
.put("xpack.security.audit.outputs", outputs)
.build();
}
@Override
protected Set<String> excludeTemplates() {
return Sets.newHashSet(SecurityIndexManager.SECURITY_TEMPLATE_NAME, IndexAuditTrail.INDEX_TEMPLATE_NAME);
}
@Override
protected int numberOfShards() {
return 1; // limit ourselves to a single shard in order to avoid timeout issues with large numbers of shards in tests
}
@Before
public void startRemoteCluster() throws IOException, InterruptedException {
final List<String> addresses = new ArrayList<>();
// get addresses for current cluster
NodesInfoResponse response = client().admin().cluster().prepareNodesInfo().execute().actionGet();
final String clusterName = response.getClusterName().value();
for (NodeInfo nodeInfo : response.getNodes()) {
TransportAddress address = nodeInfo.getTransport().address().publishAddress();
addresses.add(address.address().getHostString() + ":" + address.address().getPort());
}
// create another cluster
String cluster2Name = clusterName(Scope.TEST.name(), randomLong());
// Setup a second test cluster with a single node, security enabled, and SSL
final int numNodes = 1;
SecuritySettingsSource cluster2SettingsSource =
new SecuritySettingsSource(sslEnabled, createTempDir(), Scope.TEST) {
@Override
public Settings nodeSettings(int nodeOrdinal) {
Settings.Builder builder = Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
// Disable native ML autodetect_process as the c++ controller won't be available
// .put(MachineLearningField.AUTODETECT_PROCESS.getKey(), false)
.put("xpack.security.audit.enabled", true)
.put("xpack.security.audit.outputs", randomFrom("index", "index,logfile"))
.putList("xpack.security.audit.index.client.hosts", addresses.toArray(new String[addresses.size()]))
.put("xpack.security.audit.index.client.cluster.name", clusterName)
.put("xpack.security.audit.index.client.xpack.security.user",
TEST_USER_NAME + ":" + SecuritySettingsSourceField.TEST_PASSWORD)
.put("xpack.security.audit.index.settings.index.number_of_shards", 1)
.put("xpack.security.audit.index.settings.index.number_of_replicas", 0);
addClientSSLSettings(builder, "xpack.security.audit.index.client.xpack.security.transport.");
builder.put("xpack.security.audit.index.client.xpack.security.transport.ssl.enabled", sslEnabled);
return builder.build();
}
};
remoteCluster = new InternalTestCluster(randomLong(), createTempDir(), false, true, numNodes, numNodes,
cluster2Name, cluster2SettingsSource, 0, SECOND_CLUSTER_NODE_PREFIX, getMockPlugins(), getClientWrapper());
remoteCluster.beforeTest(random(), 0.0);
assertNoTimeout(remoteCluster.client().admin().cluster().prepareHealth().setWaitForGreenStatus().get());
}
@After
public void stopRemoteCluster() throws Exception {
List<Closeable> toStop = new ArrayList<>();
// stop the index audit trail so that the shards aren't locked causing the test to fail
toStop.add(() -> StreamSupport.stream(internalCluster().getInstances(AuditTrailService.class).spliterator(), false)
.map(s -> s.getAuditTrails()).flatMap(List::stream)
.filter(t -> t.name().equals(IndexAuditTrail.NAME))
.forEach((auditTrail) -> ((IndexAuditTrail) auditTrail).stop()));
// first stop both audit trails otherwise we keep on indexing
if (remoteCluster != null) {
toStop.add(() -> StreamSupport.stream(remoteCluster.getInstances(AuditTrailService.class).spliterator(), false)
.map(s -> s.getAuditTrails()).flatMap(List::stream)
.filter(t -> t.name().equals(IndexAuditTrail.NAME))
.forEach((auditTrail) -> ((IndexAuditTrail) auditTrail).stop()));
toStop.add(() -> remoteCluster.wipe(excludeTemplates()));
toStop.add(remoteCluster::afterTest);
toStop.add(remoteCluster);
}
IOUtils.close(toStop);
}
public void testThatRemoteAuditInstancesAreStarted() throws Exception {
logger.info("Test configuration: ssl=[{}] localAudit=[{}][{}]", sslEnabled, localAudit, outputs);
// we ensure that all instances present are started otherwise we will have issues
// and race with the shutdown logic
for (InternalTestCluster cluster : Arrays.asList(remoteCluster, internalCluster())) {
for (AuditTrailService auditTrailService : cluster.getInstances(AuditTrailService.class)) {
Optional<AuditTrail> auditTrail = auditTrailService.getAuditTrails().stream()
.filter(t -> t.name().equals(IndexAuditTrail.NAME)).findAny();
if (cluster == remoteCluster || (localAudit && outputs.contains("index"))) {
// remote cluster must be present and only if we do local audit and output to an index we are good on the local one
// as well.
assertTrue(auditTrail.isPresent());
}
if (auditTrail.isPresent()) {
IndexAuditTrail indexAuditTrail = (IndexAuditTrail) auditTrail.get();
assertBusy(() -> assertSame("trail not started remoteCluster: " + (remoteCluster == cluster),
indexAuditTrail.state(), IndexAuditTrail.State.STARTED));
}
}
}
}
}

1
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/logfile/AuditTrailSettingsUpdateTests.java
View File

@ -62,7 +62,6 @@ public class AuditTrailSettingsUpdateTests extends SecurityIntegTestCase {
// enable auditing
settingsBuilder.put("xpack.security.audit.enabled", "true");
settingsBuilder.put("xpack.security.audit.outputs", "logfile");
// add only startup filter policies
settingsBuilder.put(startupFilterSettings);
return settingsBuilder.build();

2
x-pack/plugin/sql/qa/security/build.gradle
View File

@ -29,7 +29,6 @@ subprojects {
integTestCluster {
// Setup auditing so we can use it in some tests
setting 'xpack.security.audit.enabled', 'true'
setting 'xpack.security.audit.outputs', 'logfile'
setting 'xpack.security.enabled', 'true'
setting 'xpack.license.self_generated.type', 'trial'
// Setup roles used by tests
@ -49,7 +48,6 @@ subprojects {
runqa {
// Setup auditing so we can use it in some tests
setting 'xpack.security.audit.enabled', 'true'
setting 'xpack.security.audit.outputs', 'logfile'
setting 'xpack.security.enabled', 'true'
setting 'xpack.license.self_generated.type', 'trial'
// Setup roles used by tests

40
x-pack/qa/audit-tests/build.gradle
View File

@ -1,40 +0,0 @@
apply plugin: 'elasticsearch.standalone-rest-test'
apply plugin: 'elasticsearch.rest-test'
dependencies {
testCompile project(path: xpackModule('security'), configuration: 'testArtifacts')
}
String outputDir = "${buildDir}/generated-resources/${project.name}"
task copyXPackPluginProps(type: Copy) { // wth is this?
from project(xpackModule('core')).file('src/main/plugin-metadata')
from project(xpackModule('core')).tasks.pluginProperties
from project(xpackModule('security')).file('src/main/plugin-metadata')
from project(xpackModule('security')).tasks.pluginProperties
into outputDir
}
project.sourceSets.test.output.dir(outputDir, builtBy: copyXPackPluginProps)
integTestCluster {
distribution 'default'
setting 'xpack.ilm.enabled', 'false'
setting 'xpack.ml.enabled', 'false'
setting 'xpack.monitoring.enabled', 'false'
setting 'xpack.security.enabled', 'true'
setting 'xpack.security.audit.enabled', 'true'
setting 'xpack.security.audit.outputs', 'index'
setting 'xpack.license.self_generated.type', 'trial'
setting 'logger.level', 'DEBUG'
setupCommand 'setupDummyUser',
'bin/elasticsearch-users', 'useradd', 'test_user', '-p', 'x-pack-test-password', '-r', 'superuser'
waitCondition = { node, ant ->
File tmpFile = new File(node.cwd, 'wait.success')
ant.get(src: "http://${node.httpUri()}/_cluster/health?wait_for_nodes=>=${numNodes}&wait_for_status=yellow",
dest: tmpFile.toString(),
username: 'test_user',
password: 'x-pack-test-password',
ignoreerrors: true,
retries: 10)
return tmpFile.exists()
}
}

226
x-pack/qa/audit-tests/src/test/java/org/elasticsearch/xpack/security/audit/IndexAuditIT.java
View File

@ -1,226 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.security.audit;
import com.carrotsearch.hppc.cursors.ObjectCursor;
import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.Response;
import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
import org.elasticsearch.common.io.stream.NamedWriteableRegistry;
import org.elasticsearch.common.network.NetworkModule;
import org.elasticsearch.common.settings.SecureString;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.tasks.Task;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.TestCluster;
import org.elasticsearch.xpack.core.XPackClientPlugin;
import org.elasticsearch.xpack.core.security.SecurityField;
import org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken;
import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.Arrays;
import java.util.Collection;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.Matchers.is;
public class IndexAuditIT extends ESIntegTestCase {
private static final String USER = "test_user";
private static final String PASS = "x-pack-test-password";
@Override
protected TestCluster buildTestCluster(Scope scope, long seed) throws IOException {
TestCluster testCluster = super.buildTestCluster(scope, seed);
return new TestCluster(seed) {
@Override
public void afterTest() throws IOException {
testCluster.afterTest();
}
@Override
public Client client() {
return testCluster.client();
}
@Override
public int size() {
return testCluster.size();
}
@Override
public int numDataNodes() {
return testCluster.numDataNodes();
}
@Override
public int numDataAndMasterNodes() {
return testCluster.numDataAndMasterNodes();
}
@Override
public InetSocketAddress[] httpAddresses() {
return testCluster.httpAddresses();
}
@Override
public void close() throws IOException {
testCluster.close();
}
@Override
public void ensureEstimatedStats() {
// stats are not going to be accurate for these tests since the index audit trail
// is running and changing the values so we wrap the test cluster to skip these
// checks
}
@Override
public String getClusterName() {
return testCluster.getClusterName();
}
@Override
public Iterable<Client> getClients() {
return testCluster.getClients();
}
@Override
public NamedWriteableRegistry getNamedWriteableRegistry() {
return testCluster.getNamedWriteableRegistry();
}
};
}
public void testIndexAuditTrailWorking() throws Exception {
Request request = new Request("GET", "/");
RequestOptions.Builder options = request.getOptions().toBuilder();
options.addHeader(UsernamePasswordToken.BASIC_AUTH_HEADER,
UsernamePasswordToken.basicAuthHeaderValue(USER, new SecureString(PASS.toCharArray())));
request.setOptions(options);
Response response = getRestClient().performRequest(request);
final AtomicReference<ClusterState> lastClusterState = new AtomicReference<>();
final boolean found = awaitSecurityAuditIndex(lastClusterState, QueryBuilders.matchQuery("principal", USER));
assertTrue("Did not find security audit index. Current cluster state:\n" + lastClusterState.get().toString(), found);
SearchResponse searchResponse = client().prepareSearch(".security_audit_log*").setQuery(
QueryBuilders.matchQuery("principal", USER)).get();
assertThat(searchResponse.getHits().getHits().length, greaterThan(0));
assertThat(searchResponse.getHits().getAt(0).getSourceAsMap().get("principal"), is(USER));
}
public void testAuditTrailTemplateIsRecreatedAfterDelete() throws Exception {
// this is already "tested" by the test framework since we wipe the templates before and after,
// but lets be explicit about the behavior
awaitIndexTemplateCreation();
// delete the template
AcknowledgedResponse deleteResponse = client().admin().indices()
.prepareDeleteTemplate(IndexAuditTrail.INDEX_TEMPLATE_NAME).execute().actionGet();
assertThat(deleteResponse.isAcknowledged(), is(true));
awaitIndexTemplateCreation();
}
public void testOpaqueIdWorking() throws Exception {
Request request = new Request("GET", "/");
RequestOptions.Builder options = request.getOptions().toBuilder();
options.addHeader(Task.X_OPAQUE_ID, "foo");
options.addHeader(UsernamePasswordToken.BASIC_AUTH_HEADER,
UsernamePasswordToken.basicAuthHeaderValue(USER, new SecureString(PASS.toCharArray())));
request.setOptions(options);
Response response = getRestClient().performRequest(request);
assertThat(response.getStatusLine().getStatusCode(), is(200));
final AtomicReference<ClusterState> lastClusterState = new AtomicReference<>();
final boolean found = awaitSecurityAuditIndex(lastClusterState, QueryBuilders.matchQuery("opaque_id", "foo"));
assertTrue("Did not find security audit index. Current cluster state:\n" + lastClusterState.get().toString(), found);
SearchResponse searchResponse = client().prepareSearch(".security_audit_log*").setQuery(
QueryBuilders.matchQuery("opaque_id", "foo")).get();
assertThat(searchResponse.getHits().getHits().length, greaterThan(0));
assertThat(searchResponse.getHits().getAt(0).getSourceAsMap().get("opaque_id"), is("foo"));
}
private boolean awaitSecurityAuditIndex(AtomicReference<ClusterState> lastClusterState,
QueryBuilder query) throws InterruptedException {
final AtomicBoolean indexExists = new AtomicBoolean(false);
return awaitBusy(() -> {
if (indexExists.get() == false) {
ClusterState state = client().admin().cluster().prepareState().get().getState();
lastClusterState.set(state);
for (ObjectCursor<String> cursor : state.getMetaData().getIndices().keys()) {
if (cursor.value.startsWith(".security_audit_log")) {
logger.info("found audit index [{}]", cursor.value);
indexExists.set(true);
break;
}
}
if (indexExists.get() == false) {
return false;
}
}
ensureYellowAndNoInitializingShards(".security_audit_log*");
logger.info("security audit log index is yellow");
ClusterState state = client().admin().cluster().prepareState().get().getState();
lastClusterState.set(state);
logger.info("refreshing audit indices");
client().admin().indices().prepareRefresh(".security_audit_log*").get();
logger.info("refreshed audit indices");
return client().prepareSearch(".security_audit_log*").setQuery(query)
.get().getHits().getTotalHits().value > 0;
}, 60L, TimeUnit.SECONDS);
}
private void awaitIndexTemplateCreation() throws InterruptedException {
boolean found = awaitBusy(() -> {
GetIndexTemplatesResponse response = client().admin().indices()
.prepareGetTemplates(IndexAuditTrail.INDEX_TEMPLATE_NAME).execute().actionGet();
if (response.getIndexTemplates().size() > 0) {
for (IndexTemplateMetaData indexTemplateMetaData : response.getIndexTemplates()) {
if (IndexAuditTrail.INDEX_TEMPLATE_NAME.equals(indexTemplateMetaData.name())) {
return true;
}
}
}
return false;
});
assertThat("index template [" + IndexAuditTrail.INDEX_TEMPLATE_NAME + "] was not created", found, is(true));
}
@Override
protected Settings externalClusterClientSettings() {
return Settings.builder()
.put(SecurityField.USER_SETTING.getKey(), USER + ":" + PASS)
.put(NetworkModule.TRANSPORT_TYPE_KEY, "security4")
.build();
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return Arrays.asList(XPackClientPlugin.class);
}
}

2
x-pack/qa/rolling-upgrade/build.gradle
View File

@ -156,7 +156,6 @@ subprojects {
setting 'xpack.security.transport.ssl.enabled', 'true'
setting 'xpack.security.authc.token.enabled', 'true'
setting 'xpack.security.audit.enabled', 'true'
setting 'xpack.security.audit.outputs', 'index'
setting 'xpack.security.transport.ssl.keystore.path', 'testnode.jks'
setting 'xpack.security.transport.ssl.keystore.password', 'testnode'
dependsOn copyTestNodeKeystore
@ -235,7 +234,6 @@ subprojects {
setting 'node.attr.upgraded', 'true'
setting 'xpack.security.authc.token.enabled', 'true'
setting 'xpack.security.audit.enabled', 'true'
setting 'xpack.security.audit.outputs', 'index'
setting 'node.name', "upgraded-node-${stopNode}"
dependsOn copyTestNodeKeystore
extraConfigFile 'testnode.jks', new File(outputDir + '/testnode.jks')

97
x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/IndexAuditUpgradeIT.java
View File

@ -1,97 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.upgrades;
import org.elasticsearch.Version;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.common.Booleans;
import org.hamcrest.Matchers;
import java.io.IOException;
import java.util.List;
import java.util.Map;
import static org.hamcrest.Matchers.hasSize;
public class IndexAuditUpgradeIT extends AbstractUpgradeTestCase {
public void testAuditLogs() throws Exception {
assertBusy(() -> {
assertAuditDocsExist();
assertNumUniqueNodeNameBuckets(expectedNumUniqueNodeNameBuckets());
});
}
private int expectedNumUniqueNodeNameBuckets() throws IOException {
switch (CLUSTER_TYPE) {
case OLD:
// There are three nodes in the initial test cluster
return 3;
case MIXED:
if (false == masterIsNewVersion()) {
return 3;
}
if (Booleans.parseBoolean(System.getProperty("tests.first_round"))) {
// One of the old nodes has been removed and we've added a new node
return 4;
}
// Two of the old nodes have been removed and we've added two new nodes
return 5;
case UPGRADED:
return 6;
default:
throw new IllegalArgumentException("Unsupported cluster type [" + CLUSTER_TYPE + "]");
}
}
private void assertAuditDocsExist() throws Exception {
Response response = client().performRequest(new Request("GET", "/.security_audit_log*/_count"));
assertEquals(200, response.getStatusLine().getStatusCode());
Map<String, Object> responseMap = entityAsMap(response);
assertNotNull(responseMap.get("count"));
assertThat((Integer) responseMap.get("count"), Matchers.greaterThanOrEqualTo(1));
}
private void assertNumUniqueNodeNameBuckets(int numBuckets) throws Exception {
// call API that will hit all nodes
Map<?, ?> nodesResponse = entityAsMap(client().performRequest(new Request("GET", "/_nodes/_all/info/version")));
logger.info("all nodes {}", nodesResponse);
Request aggRequest = new Request("GET", "/.security_audit_log*/_search");
aggRequest.setJsonEntity(
"{\n" +
" \"aggs\" : {\n" +
" \"nodes\" : {\n" +
" \"terms\" : { \"field\" : \"node_name\" }\n" +
" }\n" +
" }\n" +
"}");
aggRequest.addParameter("pretty", "true");
Response aggResponse = client().performRequest(aggRequest);
Map<String, Object> aggResponseMap = entityAsMap(aggResponse);
logger.debug("aggResponse {}", aggResponseMap);
Map<?, ?> aggregations = (Map<?, ?>) aggResponseMap.get("aggregations");
assertNotNull(aggregations);
Map<?, ?> nodesAgg = (Map<?, ?>) aggregations.get("nodes");
assertNotNull(nodesAgg);
List<?> buckets = (List<?>) nodesAgg.get("buckets");
assertNotNull(buckets);
assertThat("Found node buckets " + buckets, buckets, hasSize(numBuckets));
}
/**
* Has the master been upgraded to the new version?
*/
private boolean masterIsNewVersion() throws IOException {
Map<?, ?> map = entityAsMap(client().performRequest(new Request("GET", "/_nodes/_master")));
map = (Map<?, ?>) map.get("nodes");
assertThat(map.values(), hasSize(1));
map = (Map<?, ?>) map.values().iterator().next();
Version masterVersion = Version.fromString(map.get("version").toString());
return Version.CURRENT.equals(masterVersion);
}
}