475 lines
22 KiB
Plaintext
475 lines
22 KiB
Plaintext
[[auditing]]
|
|
== Auditing Security Events
|
|
|
|
You can enable auditing to keep track of security-related events such as
|
|
authentication failures and refused connections. Logging these events enables you
|
|
to monitor your cluster for suspicious activity and provides evidence in the
|
|
event of an attack.
|
|
|
|
[IMPORTANT]
|
|
============================================================================
|
|
Audit logs are **disabled** by default. To enable this functionality, you
|
|
must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`.
|
|
============================================================================
|
|
|
|
{Security} provides two ways to persist audit logs:
|
|
|
|
* The <<audit-log-output, `logfile`>> output, which persists events to
|
|
a dedicated `<clustername>_access.log` file on the host's file system.
|
|
* The <<audit-index, `index`>> output, which persists events to an Elasticsearch index.
|
|
The audit index can reside on the same cluster, or a separate cluster.
|
|
|
|
By default, only the `logfile` output is used when enabling auditing.
|
|
To facilitate browsing and analyzing the events, you can also enable
|
|
indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`:
|
|
|
|
[source,yaml]
|
|
----------------------------
|
|
xpack.security.audit.outputs: [ index, logfile ]
|
|
----------------------------
|
|
|
|
The `index` output type should be used in conjunction with the `logfile`
|
|
output type Because it is possible for the `index` output type to lose
|
|
messages if the target index is unavailable, the `access.log` should be
|
|
used as the official record of events.
|
|
|
|
NOTE: Audit events are batched for indexing so there is a lag before
|
|
events appear in the index. You can control how frequently batches of
|
|
events are pushed to the index by setting
|
|
`xpack.security.audit.index.flush_interval` in `elasticsearch.yml`.
|
|
|
|
[float]
|
|
[[audit-event-types]]
|
|
=== Audit Event Types
|
|
|
|
Each request may generate multiple audit events.
|
|
The following is a list of the events that can be generated:
|
|
|
|
|======
|
|
| `anonymous_access_denied` | | | Logged when a request is denied due to a missing
|
|
authentication token.
|
|
| `authentication_success` | | | Logged when a user successfully authenticates.
|
|
| `authentication_failed` | | | Logged when the authentication token cannot be
|
|
matched to a known user.
|
|
| `realm_authentication_failed` | | | Logged for every realm that fails to present a valid
|
|
authentication token. `<realm>` represents the
|
|
realm type.
|
|
| `access_denied` | | | Logged when an authenticated user attempts to execute
|
|
an action they do not have the necessary
|
|
<<security-reference, privilege>> to perform.
|
|
| `access_granted` | | | Logged when an authenticated user attempts to execute
|
|
an action they have the necessary privilege to perform.
|
|
When the `system_access_granted` event is included, all system
|
|
(internal) actions are also logged. The default setting does
|
|
not log system actions to avoid cluttering the logs.
|
|
| `run_as_granted` | | | Logged when an authenticated user attempts to <<run-as-privilege, run as>>
|
|
another user that they have the necessary privileges to do.
|
|
| `run_as_denied` | | | Logged when an authenticated user attempts to <<run-as-privilege, run as>>
|
|
another user action they do not have the necessary
|
|
<<security-reference, privilege>> to do so.
|
|
| `tampered_request` | | | Logged when {security} detects that the request has
|
|
been tampered with. Typically relates to `search/scroll`
|
|
requests when the scroll ID is believed to have been
|
|
tampered with.
|
|
| `connection_granted` | | | Logged when an incoming TCP connection passes the
|
|
<<ip-filtering, IP Filter>> for a specific
|
|
profile.
|
|
| `connection_denied` | | | Logged when an incoming TCP connection does not pass the
|
|
<<ip-filtering, IP Filter>> for a specific
|
|
profile.
|
|
|======
|
|
|
|
[float]
|
|
[[audit-event-attributes]]
|
|
=== Audit Event Attributes
|
|
|
|
The following table shows the common attributes that can be associated with every event.
|
|
|
|
.Common Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `timestamp` | When the event occurred.
|
|
| `node_name` | The name of the node.
|
|
| `node_host_name` | The hostname of the node.
|
|
| `node_host_address` | The IP address of the node.
|
|
| `layer` | The layer from which this event originated: `rest`, `transport` or `ip_filter`
|
|
| `event_type` | The type of event that occurred: `anonymous_access_denied`,
|
|
`authentication_failed`, `access_denied`, `access_granted`,
|
|
`connection_granted`, `connection_denied`, `tampered_request`,
|
|
`run_as_granted`, `run_as_denied`.
|
|
|======
|
|
|
|
The following tables show the attributes that can be associated with each type of event.
|
|
The log level determines which attributes are included in a log entry.
|
|
|
|
.REST anonymous_access_denied Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `uri` | The REST endpoint URI.
|
|
| `request_body` | The body of the request, if enabled.
|
|
|======
|
|
|
|
.REST authentication_success Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `user` | The authenticated user.
|
|
| `realm` | The realm that authenticated the user.
|
|
| `uri` | The REST endpoint URI.
|
|
| `params` | The REST URI query parameters.
|
|
| `request_body` | The body of the request, if enabled.
|
|
|======
|
|
|
|
.REST authentication_failed Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `principal` | The principal (username) that failed authentication.
|
|
| `uri` | The REST endpoint URI.
|
|
| `request_body` | The body of the request, if enabled.
|
|
|======
|
|
|
|
.REST realm_authentication_failed Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `principal` | The principal (username) that failed authentication.
|
|
| `uri` | The REST endpoint URI.
|
|
| `request_body` | The body of the request, if enabled.
|
|
| `realm` | The realm that failed to authenticate the user.
|
|
NOTE: A separate entry is logged for each
|
|
consulted realm.
|
|
|======
|
|
|
|
.Transport anonymous_access_denied Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_type` | Where the request originated: `rest` (request
|
|
originated from a REST API request), `transport`
|
|
(request was received on the transport channel),
|
|
`local_node` (the local node issued the request).
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `action` | The name of the action that was executed.
|
|
| `request` | The type of request that was executed.
|
|
| `indices` | A comma-separated list of indices this request
|
|
pertains to (when applicable).
|
|
|======
|
|
|
|
.Transport authentication_success Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_type` | Where the request originated: `rest` (request
|
|
originated from a REST API request), `transport`
|
|
(request was received on the transport channel),
|
|
`local_node` (the local node issued the request).
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `user` | The authenticated user.
|
|
| `realm` | The realm that authenticated the user.
|
|
| `action` | The name of the action that was executed.
|
|
| `request` | The type of request that was executed.
|
|
|======
|
|
|
|
.Transport authentication_failed Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_type` | Where the request originated: `rest` (request
|
|
originated from a REST API request), `transport`
|
|
(request was received on the transport channel),
|
|
`local_node` (the local node issued the request).
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `principal` | The principal (username) that failed authentication.
|
|
| `action` | The name of the action that was executed.
|
|
| `request` | The type of request that was executed.
|
|
| `indices` | A comma-separated list of indices this request
|
|
pertains to (when applicable).
|
|
|======
|
|
|
|
.Transport realm_authentication_failed Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_type` | Where the request originated: `rest` (request
|
|
originated from a REST API request), `transport`
|
|
(request was received on the transport channel),
|
|
`local_node` (the local node issued the request).
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `principal` | The principal (username) that failed authentication.
|
|
| `action` | The name of the action that was executed.
|
|
| `request` | The type of request that was executed.
|
|
| `indices` | A comma-separated list of indices this request
|
|
pertains to (when applicable).
|
|
| `realm` | The realm that failed to authenticate the user.
|
|
NOTE: A separate entry is logged for each
|
|
consulted realm.
|
|
|======
|
|
|
|
.Transport access_granted Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_type` | Where the request originated: `rest` (request
|
|
originated from a REST API request), `transport`
|
|
(request was received on the transport channel),
|
|
`local_node` (the local node issued the request).
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `principal` | The principal (username) that passed authentication.
|
|
| `roles` | The set of roles granting permissions.
|
|
| `action` | The name of the action that was executed.
|
|
| `request` | The type of request that was executed.
|
|
| `indices` | A comma-separated list of indices this request
|
|
pertains to (when applicable).
|
|
|======
|
|
|
|
.Transport access_denied Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_type` | Where the request originated: `rest` (request
|
|
originated from a REST API request), `transport`
|
|
(request was received on the transport channel),
|
|
`local_node` (the local node issued the request).
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `principal` | The principal (username) that failed authentication.
|
|
| `roles` | The set of roles granting permissions.
|
|
| `action` | The name of the action that was executed.
|
|
| `request` | The type of request that was executed.
|
|
| `indices` | A comma-separated list of indices this request
|
|
relates to (when applicable).
|
|
|======
|
|
|
|
.Transport tampered_request Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_type` | Where the request originated: `rest` (request
|
|
originated from a REST API request), `transport`
|
|
(request was received on the transport channel),
|
|
`local_node` (the local node issued the request).
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `principal` | The principal (username) that failed to authenticate.
|
|
| `action` | The name of the action that was executed.
|
|
| `request` | The type of request that was executed.
|
|
| `indices` | A comma-separated list of indices this request
|
|
pertains to (when applicable).
|
|
|======
|
|
|
|
.IP Filter connection_granted Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `transport_profile` | The transport profile the request targeted.
|
|
| `rule` | The <<ip-filtering, IP filtering>> rule that granted
|
|
the request.
|
|
|======
|
|
|
|
.IP Filter connection_denied Attributes
|
|
[cols="2,7",options="header"]
|
|
|======
|
|
| Attribute | Description
|
|
| `origin_address` | The IP address from which the request originated.
|
|
| `transport_profile` | The transport profile the request targeted.
|
|
| `rule` | The <<ip-filtering, IP filtering>> rule that denied
|
|
the request.
|
|
|======
|
|
|
|
[float]
|
|
[[audit-log-output]]
|
|
=== Logfile Audit Output
|
|
|
|
The `logfile` audit output is the default output for auditing. It writes data to
|
|
the `<clustername>_access.log` file in the logs directory.
|
|
|
|
[float]
|
|
[[audit-log-entry-format]]
|
|
=== Log Entry Format
|
|
|
|
The format of a log entry is:
|
|
|
|
[source,txt]
|
|
----------------------------------------------------------------------------
|
|
[<timestamp>] [<local_node_info>] [<layer>] [<entry_type>] <attribute_list>
|
|
----------------------------------------------------------------------------
|
|
|
|
`<timestamp>` :: When the event occurred. You can configure the
|
|
timestamp format in `log4j2.properties`.
|
|
`<local_node_info>` :: Information about the local node that generated
|
|
the log entry. You can control what node information
|
|
is included by configuring the
|
|
{ref}/auditing-settings.html#node-audit-settings[local node info settings].
|
|
`<layer>` :: The layer from which this event originated:
|
|
`rest`, `transport` or `ip_filter`.
|
|
`<entry_type>` :: The type of event that occurred: `anonymous_access_denied`,
|
|
`authentication_failed`, `access_denied`, `access_granted`,
|
|
`connection_granted`, `connection_denied`.
|
|
`<attribute_list>` :: A comma-separated list of key-value pairs that contain
|
|
data pertaining to the event. Formatted as
|
|
`attr1=[val1], attr2=[val2]`. See <<audit-event-attributes,
|
|
Audit Entry Attributes>> for the attributes that can be included
|
|
for each type of event.
|
|
|
|
[float]
|
|
[[audit-log-settings]]
|
|
=== Logfile Output Settings
|
|
|
|
The events and some other information about what gets logged can be
|
|
controlled using settings in the `elasticsearch.yml` file. See
|
|
{ref}/auditing-settings.html#event-audit-settings[Audited Event Settings] and
|
|
{ref}/auditing-settings.html#node-audit-settings[Local Node Info Settings].
|
|
|
|
IMPORTANT: No filtering is performed when auditing, so sensitive data may be
|
|
audited in plain text when including the request body in audit events.
|
|
|
|
[[logging-file]]
|
|
You can also configure how the logfile is written in the `log4j2.properties`
|
|
file located in `CONFIG_DIR/x-pack`. By default, audit information is appended to the
|
|
`<clustername>_access.log` file located in the standard Elasticsearch `logs` directory
|
|
(typically located at `$ES_HOME/logs`). The file rolls over on a daily basis.
|
|
|
|
[float]
|
|
[[audit-log-ignore-policy]]
|
|
=== Logfile Audit Events Ignore Policies
|
|
|
|
The comprehensive audit trail is necessary to ensure accountability. It offers tremendous
|
|
value during incident response and can even be required for demonstrating compliance.
|
|
|
|
The drawback of an audited system is represented by the inevitable performance penalty incurred.
|
|
In all truth, the audit trail spends _I/O ops_ that are not available anymore for the user's queries.
|
|
Sometimes the verbosity of the audit trail may become a problem that the event type restrictions,
|
|
<<audit-log-settings, defined by `include` and `exclude`>>, will not alleviate.
|
|
|
|
*Audit events ignore policies* are a finer way to tune the verbosity of the audit trail.
|
|
These policies define rules that match audit events which will be _ignored_ (read as: not printed).
|
|
Rules match on the values of attributes of audit events and complement the <<audit-log-settings, include/exclude>> method.
|
|
Imagine the corpus of audit events and the policies chopping off unwanted events.
|
|
|
|
IMPORTANT: When utilizing audit events ignore policies you are acknowledging potential
|
|
accountability gaps that could render illegitimate actions undetectable.
|
|
Please take time to review these policies whenever your system architecture changes.
|
|
|
|
A policy is a named set of filter rules. Each filter rule applies to a single event attribute,
|
|
one of the `users`, `realms`, `roles` or `indices` attributes. The filter rule defines
|
|
a list of {ref}/query-dsl-regexp-query.html#regexp-syntax[Lucene regexp], *any* of which has to match the value of the audit
|
|
event attribute for the rule to match.
|
|
A policy matches an event if *all* the rules comprising it match the event.
|
|
An audit event is ignored, therefore not printed, if it matches *any* policy. All other
|
|
non-matching events are printed as usual.
|
|
|
|
All policies are defined under the `xpack.security.audit.logfile.events.ignore_filters`
|
|
settings namespace. For example, the following policy named _example1_ matches
|
|
events from the _kibana_ or _admin_user_ principals **and** operating over indices of the
|
|
wildcard form _app-logs*_:
|
|
|
|
[source,yaml]
|
|
----------------------------
|
|
xpack.security.audit.logfile.events.ignore_filters:
|
|
example1:
|
|
users: ["kibana", "admin_user"]
|
|
indices: ["app-logs*"]
|
|
----------------------------
|
|
|
|
An audit event generated by the _kibana_ user and operating over multiple indices
|
|
, some of which do not match the indices wildcard, will not match.
|
|
As expected, operations generated by all other users (even operating only on indices that
|
|
match the _indices_ filter) will not match this policy either.
|
|
|
|
Audit events of different types may have <<audit-event-attributes, different attributes>>.
|
|
If an event does not contain an attribute for which some policy defines filters, the
|
|
event will not match the policy.
|
|
For example, the following policy named _example2_, will never match `authentication_success` or
|
|
`authentication_failed` events, irrespective of the user's roles, because these
|
|
event schemas do not contain the `role` attribute:
|
|
|
|
[source,yaml]
|
|
----------------------------
|
|
xpack.security.audit.logfile.events.ignore_filters:
|
|
example2:
|
|
roles: ["admin", "ops_admin_*"]
|
|
----------------------------
|
|
|
|
Likewise, any events of users with multiple roles, some of which do not match the
|
|
regexps will not match this policy.
|
|
|
|
For completeness, although practical use cases should be sparse, a filter can match
|
|
a missing attribute of an event, using the empty string ("") or the empty list ([]).
|
|
For example, the following policy will match events that do not have the `indices`
|
|
attribute (`anonymous_access_denied`, `authentication_success` and other types) as well
|
|
as events over the _next_ index.
|
|
|
|
[source,yaml]
|
|
----------------------------
|
|
xpack.security.audit.logfile.events.ignore_filters:
|
|
example3:
|
|
indices: ["next", ""]
|
|
----------------------------
|
|
|
|
|
|
[float]
|
|
[[audit-index]]
|
|
=== Index Audit Output
|
|
|
|
In addition to logging to a file, you can store audit logs in Elasticsearch
|
|
rolling indices. These indices can be either on the same cluster, or on a
|
|
remote cluster. You configure the following settings in
|
|
`elasticsearch.yml` to control how audit entries are indexed. To enable
|
|
this output, you need to configure the setting `xpack.security.audit.outputs`
|
|
in the `elasticsearch.yml` file:
|
|
|
|
[source,yaml]
|
|
----------------------------
|
|
xpack.security.audit.outputs: [ index, logfile ]
|
|
----------------------------
|
|
|
|
For more configuration options, see
|
|
{ref}/auditing-settings.html#index-audit-settings[Audit Log Indexing Configuration Settings].
|
|
|
|
IMPORTANT: No filtering is performed when auditing, so sensitive data may be
|
|
audited in plain text when including the request body in audit events.
|
|
|
|
[float]
|
|
==== Audit Index Settings
|
|
|
|
You can also configure settings for the indices that the events are stored in.
|
|
These settings are configured in the `xpack.security.audit.index.settings` namespace
|
|
in `elasticsearch.yml`. For example, the following configuration sets the
|
|
number of shards and replicas to 1 for the audit indices:
|
|
|
|
[source,yaml]
|
|
----------------------------
|
|
xpack.security.audit.index.settings:
|
|
index:
|
|
number_of_shards: 1
|
|
number_of_replicas: 1
|
|
----------------------------
|
|
|
|
[float]
|
|
==== Forwarding Audit Logs to a Remote Cluster
|
|
|
|
To index audit events to a remote Elasticsearch cluster, you configure
|
|
the following `xpack.security.audit.index.client` settings:
|
|
|
|
* `xpack.security.audit.index.client.hosts`
|
|
* `xpack.security.audit.index.client.cluster.name`
|
|
* `xpack.security.audit.index.client.xpack.security.user`
|
|
|
|
For more information about these settings, see
|
|
{ref}/auditing-settings.html#remote-audit-settings[Remote Audit Log Indexing Configuration Settings].
|
|
|
|
You can pass additional settings to the remote client by specifying them in the
|
|
`xpack.security.audit.index.client` namespace. For example, to allow the remote
|
|
client to discover all of the nodes in the remote cluster you can specify the
|
|
`client.transport.sniff` setting:
|
|
|
|
[source,yaml]
|
|
----------------------------
|
|
xpack.security.audit.index.client.transport.sniff: true
|
|
----------------------------
|