Elasticsearch support to JSON logging (#36833)
In order to support JSON log format, a custom pattern layout was used and its configuration is enclosed in ESJsonLayout. Users are free to use their own patterns, but if smooth Beats integration is needed, they should use ESJsonLayout. EvilLoggerTests are left intact to make sure user's custom log patterns work fine.
To populate additional fields node.id and cluster.uuid which are not available at start time,
a cluster state update will have to be received and the values passed to log4j pattern converter.
A ClusterStateObserver.Listener is used to receive only one ClusteStateUpdate. Once update is received the nodeId and clusterUUid are set in a static field in a NodeAndClusterIdConverter.
Following fields are expected in JSON log lines: type, tiemstamp, level, component, cluster.name, node.name, node.id, cluster.uuid, message, stacktrace
see ESJsonLayout.java for more details and field descriptions
Docker log4j2 configuration is now almost the same as the one use for ES binary.
The only difference is that docker is using console appenders, whereas ES is using file appenders.
relates: #32850
2019-01-29 01:20:09 -05:00
|
|
|
[float]
|
|
|
|
|
[[breaking_70_logging_changes]]
|
|
|
|
|
=== Logging changes
|
|
|
|
|
|
2019-04-08 21:54:29 -04:00
|
|
|
//NOTE: The notable-breaking-changes tagged regions are re-used in the
|
|
|
|
|
//Installation and Upgrade Guide
|
|
|
|
|
|
|
|
|
|
//tag::notable-breaking-changes[]
|
|
|
|
|
|
|
|
|
|
// end::notable-breaking-changes[]
|
|
|
|
|
|
Elasticsearch support to JSON logging (#36833)
In order to support JSON log format, a custom pattern layout was used and its configuration is enclosed in ESJsonLayout. Users are free to use their own patterns, but if smooth Beats integration is needed, they should use ESJsonLayout. EvilLoggerTests are left intact to make sure user's custom log patterns work fine.
To populate additional fields node.id and cluster.uuid which are not available at start time,
a cluster state update will have to be received and the values passed to log4j pattern converter.
A ClusterStateObserver.Listener is used to receive only one ClusteStateUpdate. Once update is received the nodeId and clusterUUid are set in a static field in a NodeAndClusterIdConverter.
Following fields are expected in JSON log lines: type, tiemstamp, level, component, cluster.name, node.name, node.id, cluster.uuid, message, stacktrace
see ESJsonLayout.java for more details and field descriptions
Docker log4j2 configuration is now almost the same as the one use for ES binary.
The only difference is that docker is using console appenders, whereas ES is using file appenders.
relates: #32850
2019-01-29 01:20:09 -05:00
|
|
|
[float]
|
2019-05-28 14:16:18 -04:00
|
|
|
[[new-json-format-log-directory]]
|
Elasticsearch support to JSON logging (#36833)
In order to support JSON log format, a custom pattern layout was used and its configuration is enclosed in ESJsonLayout. Users are free to use their own patterns, but if smooth Beats integration is needed, they should use ESJsonLayout. EvilLoggerTests are left intact to make sure user's custom log patterns work fine.
To populate additional fields node.id and cluster.uuid which are not available at start time,
a cluster state update will have to be received and the values passed to log4j pattern converter.
A ClusterStateObserver.Listener is used to receive only one ClusteStateUpdate. Once update is received the nodeId and clusterUUid are set in a static field in a NodeAndClusterIdConverter.
Following fields are expected in JSON log lines: type, tiemstamp, level, component, cluster.name, node.name, node.id, cluster.uuid, message, stacktrace
see ESJsonLayout.java for more details and field descriptions
Docker log4j2 configuration is now almost the same as the one use for ES binary.
The only difference is that docker is using console appenders, whereas ES is using file appenders.
relates: #32850
2019-01-29 01:20:09 -05:00
|
|
|
==== New JSON format log files in `log` directory
|
|
|
|
|
|
|
|
|
|
Elasticsearch now will produce additional log files in JSON format. They will be stored in `*.json` suffix files.
|
|
|
|
|
Following files should be expected now in log directory:
|
|
|
|
|
* ${cluster_name}_server.json
|
|
|
|
|
* ${cluster_name}_deprecation.json
|
|
|
|
|
* ${cluster_name}_index_search_slowlog.json
|
|
|
|
|
* ${cluster_name}_index_indexing_slowlog.json
|
|
|
|
|
* ${cluster_name}.log
|
|
|
|
|
* ${cluster_name}_deprecation.log
|
|
|
|
|
* ${cluster_name}_index_search_slowlog.log
|
|
|
|
|
* ${cluster_name}_index_indexing_slowlog.log
|
|
|
|
|
* ${cluster_name}_audit.json
|
|
|
|
|
* gc.log
|
|
|
|
|
|
|
|
|
|
Note: You can configure which of these files are written by editing `log4j2.properties`.
|
|
|
|
|
|
|
|
|
|
[float]
|
2019-05-28 14:16:18 -04:00
|
|
|
[[log-files-ending-log-deprecated]]
|
Elasticsearch support to JSON logging (#36833)
In order to support JSON log format, a custom pattern layout was used and its configuration is enclosed in ESJsonLayout. Users are free to use their own patterns, but if smooth Beats integration is needed, they should use ESJsonLayout. EvilLoggerTests are left intact to make sure user's custom log patterns work fine.
To populate additional fields node.id and cluster.uuid which are not available at start time,
a cluster state update will have to be received and the values passed to log4j pattern converter.
A ClusterStateObserver.Listener is used to receive only one ClusteStateUpdate. Once update is received the nodeId and clusterUUid are set in a static field in a NodeAndClusterIdConverter.
Following fields are expected in JSON log lines: type, tiemstamp, level, component, cluster.name, node.name, node.id, cluster.uuid, message, stacktrace
see ESJsonLayout.java for more details and field descriptions
Docker log4j2 configuration is now almost the same as the one use for ES binary.
The only difference is that docker is using console appenders, whereas ES is using file appenders.
relates: #32850
2019-01-29 01:20:09 -05:00
|
|
|
==== Log files ending with `*.log` deprecated
|
|
|
|
|
Log files with the `.log` file extension using the old pattern layout format
|
|
|
|
|
are now considered deprecated and the newly added JSON log file format with
|
|
|
|
|
the `.json` file extension should be used instead.
|
|
|
|
|
Note: GC logs which are written to the file `gc.log` will not be changed.
|
|
|
|
|
|
|
|
|
|
[float]
|
|
|
|
|
==== Docker output in JSON format
|
|
|
|
|
|
|
|
|
|
All Docker console logs are now in JSON format. You can distinguish logs streams with the `type` field.
|
2019-02-05 03:09:15 -05:00
|
|
|
|
|
|
|
|
[float]
|
|
|
|
|
==== Audit plaintext log file removed, JSON file renamed
|
|
|
|
|
|
|
|
|
|
Elasticsearch no longer produces the `${cluster_name}_access.log` plaintext
|
|
|
|
|
audit log file. The `${cluster_name}_audit.log` files also no longer exist; they
|
|
|
|
|
are replaced by `${cluster_name}_audit.json` files. When auditing is enabled,
|
|
|
|
|
auditing events are stored in these dedicated JSON log files on each node.
|
|
|
|
|
|
