[Docs] audit logfile structured format (#34584)

Documents the new structured logfile format for auditing
that was introduced by #31931. Most changes herein
are for 6.x . In 7.0 the deprecated format is gone and a
follow-up PR is in order.

docs/reference/settings/audit-settings.asciidoc

@@ -18,10 +18,18 @@ Set to `true` to enable auditing on the node. The default value is `false`.
 `xpack.security.audit.outputs`::
 Specifies where audit logs are output. For example: `[ index, logfile ]`. The
 default value is `logfile`, which puts the auditing events in a dedicated
-`<clustername>_access.log` file on the node. You can also specify `index`, which
-puts the auditing events in an {es} index that is prefixed with
-`.security_audit_log`. The index can reside on the same cluster or a separate
-cluster.
+file named `<clustername>_audit.log` on each node.
+You can also specify `index`, which puts the auditing events in an {es} index
+that is prefixed with `.security_audit_log`. The index can reside on the same
+cluster or a separate cluster.
+
+For backwards compatibility reasons, if you use the logfile output type, a
+`<clustername>_access.log` file is also created. It contains the same
+information, but it uses the older (pre-6.5.0) formatting style.
+If the backwards compatible format is not required, it should be disabled.
+To do that, change its logger level to `off` in the `log4j2.properties` file.
+For more information, see <<configuring-logging-levels>>.
+
 +
 --
 TIP: If the index is unavailable, it is possible for auditing events to
@@ -57,17 +65,27 @@ audited in plain text when including the request body in audit events.
 [[node-audit-settings]]
 ==== Local Node Info Settings
 
-`xpack.security.audit.logfile.prefix.emit_node_name`::
-Specifies whether to include the node's name in the local node info. The
-default value is `true`.
+`xpack.security.audit.logfile.emit_node_name`::
+Specifies whether to include the <<node.name,node name>> as a field in
+each audit event.
+The default value is `true`.
 
-`xpack.security.audit.logfile.prefix.emit_node_host_address`::
-Specifies whether to include the node's IP address in the local node info. The
-default value is `false`.
+`xpack.security.audit.logfile.emit_node_host_address`::
+Specifies whether to include the node's IP address as a field in each audit event.
+The default value is `false`.
 
-`xpack.security.audit.logfile.prefix.emit_node_host_name`::
-Specifies whether to include the node's host name in the local node info. The
-default value is `false`.
+`xpack.security.audit.logfile.emit_node_host_name`::
+Specifies whether to include the node's host name as a field in each audit event.
+The default value is `false`.
+
+`xpack.security.audit.logfile.emit_node_id`::
+Specifies whether to include the node id as a field in each audit event.
+This is available for the new format only. That is to say, this information
+does not exist in the `<clustername>_access.log` file.
+Unlike <<node.name,node name>>, whose value might change if the administrator
+changes the setting in the config file, the node id will persist across cluster
+restarts and the administrator cannot change it.
+The default value is `true`.
 
 [[index-audit-settings]]
 ==== Audit Log Indexing Configuration Settings

x-pack/docs/en/security/auditing/output-logfile.asciidoc

@@ -3,13 +3,41 @@
 === Logfile audit output
 
 The `logfile` audit output is the default output for auditing. It writes data to
-the `<clustername>_access.log` file in the logs directory.
+the `<clustername>_audit.log` file in the logs directory. To maintain
+compatibility with releases prior to 6.5.0, a `<clustername>_access.log` file
+is also generated. They differ in the output format but the contents
+are similar. For systems that are not ingesting the audit file for search or
+analytics it is strongly recommended to only keep the newer format.
+Turning off the deprecated output format can be achieved by disabling the logger
+in the `log4j2.properties` file (hint: there is a config comment
+about it).
+For more information, see {ref}/logging.html#configuring-logging-levels[configuring-logging].
+
 
 [float]
 [[audit-log-entry-format]]
 === Log entry format
 
-The format of a log entry is:
+The log entries in the `<clustername>_audit.log` file 
+have the following format:
+
+- Each log entry is a one line JSON document and each one is printed on a separate line.
+- The fields of a log entry are ordered. However, if a field does not have a value it
+  will not be printed. The precise line pattern, together with the complete field
+  order, are specified in the `log4j2.properties` config file.
+- The log entry does not contain nested inner JSON objects, i.e. the doc is flat.
+- The field names follow a dotted notation to flatten inner objects.
+- A field's value can be a string, a number or an array of strings.
+- A field's value, a request body as well, will be escaped as per the JSON RFC 4627.
+
+There is a list of <<audit-event-types, audit event types>> specifying the
+set of fields for each sog entry type.
+
+[float]
+[[deprecated-audit-log-entry-format]]
+=== Deprecated log entry format
+
+The log entries in the `<clustername>_access.log` file have the following format:
 
 [source,txt]
 ----------------------------------------------------------------------------
@@ -48,8 +76,14 @@ audited in plain text when including the request body in audit events.
 [[logging-file]]
 You can also configure how the logfile is written in the `log4j2.properties`
 file located in `ES_PATH_CONF`. By default, audit information is appended to the
-`<clustername>_access.log` file located in the standard Elasticsearch `logs` directory
+`<clustername>_audit.log` file located in the standard Elasticsearch `logs` directory
 (typically located at `$ES_HOME/logs`). The file rolls over on a daily basis.
+The deprecated logfile audit format (`<clustername>_access.log`) can be disabled
+from the same `log4j2.properties` file (hint: look for the comment
+instructing to set the log level to `off`). The deprecated format is a duplication
+of information that is in place to assure backwards compatibility. If you are
+not strict about the audit format it is strongly recommended to only use the
+`<clustername>_audit.log` log appender.
 
 [float]
 [[audit-log-ignore-policy]]

x-pack/docs/en/security/auditing/overview.asciidoc

@@ -16,11 +16,14 @@ must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`.
 {Security} provides two ways to persist audit logs:
 
 * The <<audit-log-output, `logfile`>> output, which persists events to
-  a dedicated `<clustername>_access.log` file on the host's file system.
-* The <<audit-index, `index`>> output, which persists events to an Elasticsearch index.
-The audit index can reside on the same cluster, or a separate cluster.
+  a dedicated `<clustername>_audit.log` file on the host's file system.
+  For backwards compatibility reasons, a file named `<clustername>_access.log`
+  is also generated.
+* The <<audit-index, `index`>> output, which persists events to an Elasticsearch
+  index. The audit index can reside on the same cluster, or a separate cluster.
 
-By default, only the `logfile` output is used when enabling auditing.
+By default, only the `logfile` output is used when enabling auditing,
+implicitly outputing to both `<clustername>_audit.log` and `<clustername>_access.log`.
 To facilitate browsing and analyzing the events, you can also enable
 indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`: