Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-17 07:25:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

update log4j example (#13095)

Browse Source 
* update log4j example

* fix some style issues

* Update docs/configuration/logging.md

Co-authored-by: Frank Chen <frankchen@apache.org>

Co-authored-by: Frank Chen <frankchen@apache.org>
This commit is contained in:
Charles Smith 2022-09-21 18:46:49 -07:00 committed by GitHub
parent 12f12a13a9
commit eb760c3d1d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 81 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

123
docs/configuration/logging.md
View File

@ -23,80 +23,118 @@ title: "Logging"
--> -->
Apache Druid processes will emit logs that are useful for debugging to log files. Apache Druid services emit logs that to help you debug.
These processes also emit periodic [metrics](../configuration/index.md#enabling-metrics) about their state. The same services also emit periodic [metrics](../configuration/index.md#enabling-metrics) about their state.
Metric info logs can be disabled with `-Ddruid.emitter.logging.logLevel=debug`. To disable metric info logs set the following runtime property: `-Ddruid.emitter.logging.logLevel=debug`.
Druid uses [log4j2](http://logging.apache.org/log4j/2.x/) for logging. Druid uses [log4j2](http://logging.apache.org/log4j/2.x/) for logging.
The default configuration file log4j2.xml ships with Druid under conf/druid/{config}/_common/log4j2.xml . The default configuration file log4j2.xml ships with Druid at the following path: `conf/druid/{config}/_common/log4j2.xml`.
By default, Druid uses `RollingRandomAccessFile` for rollover daily, and keeps log files up to 7 days. By default, Druid uses `RollingRandomAccessFile` for rollover daily, and keeps log files up to 7 days.
If that's not suitable in your case, you could modify the log4j2.xml to meet your need. If that's not suitable in your case, modify the `log4j2.xml` accordingly.
An example log4j2.xml file is shown below: The following example log4j2.xml is based upon the micro quickstart:
``` ```
<?xml version="1.0" encoding="UTF-8" ?> <?xml version="1.0" encoding="UTF-8" ?>
<Configuration status="WARN"> <Configuration status="WARN">
<Console name="Console" target="SYSTEM_OUT"> <Properties>
<PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/> <!-- to change log directory, set DRUID_LOG_DIR environment variable to your directory before launching Druid -->
</Console> <Property name="druid.log.path" value="log" />
</Properties>
<RollingRandomAccessFile name="FileAppender"
fileName="${sys:druid.log.path}/${sys:druid.node.type}.log" <Appenders>
filePattern="${sys:druid.log.path}/${sys:druid.node.type}.%d{yyyyMMdd}.log"> <Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/> <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
<Policies> </Console>
<!-- Rolling Files-->
<RollingRandomAccessFile name="FileAppender"
fileName="${sys:druid.log.path}/${sys:druid.node.type}.log"
filePattern="${sys:druid.log.path}/${sys:druid.node.type}.%d{yyyyMMdd}.log">
<PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
<Policies>
<TimeBasedTriggeringPolicy interval="1" modulate="true"/> <TimeBasedTriggeringPolicy interval="1" modulate="true"/>
</Policies> </Policies>
<DefaultRolloverStrategy> <DefaultRolloverStrategy>
<Delete basePath="${sys:druid.log.path}/" maxDepth="1"> <Delete basePath="${sys:druid.log.path}/" maxDepth="1">
<IfFileName glob="*.log" /> <IfFileName glob="*.log" />
<IfLastModified age="7d" /> <IfLastModified age="7d" />
</Delete> </Delete>
</DefaultRolloverStrategy> </DefaultRolloverStrategy>
</RollingRandomAccessFile> </RollingRandomAccessFile>
</Appenders>
<Loggers> <Loggers>
<Root level="info"> <Root level="info">
<AppenderRef ref="FileAppender"/> <AppenderRef ref="FileAppender"/>
</Root> </Root>
<!-- Uncomment to enable logging of all HTTP requests <!-- Set level="debug" to see stack traces for query errors -->
<Logger name="org.apache.druid.jetty.RequestLog" additivity="false" level="DEBUG"> <Logger name="org.apache.druid.server.QueryResource" level="info" additivity="false">
<AppenderRef ref="FileAppender"/> <Appender-ref ref="FileAppender"/>
</Logger>
<Logger name="org.apache.druid.server.QueryLifecycle" level="info" additivity="false">
<Appender-ref ref="FileAppender"/>
</Logger>
<!-- Set level="debug" or "trace" to see more Coordinator details (segment balancing, load/drop rules, etc) -->
<Logger name="org.apache.druid.server.coordinator" level="info" additivity="false">
<Appender-ref ref="FileAppender"/>
</Logger>
<!-- Set level="debug" to see low-level details about segments and ingestion -->
<Logger name="org.apache.druid.segment" level="info" additivity="false">
<Appender-ref ref="FileAppender"/>
</Logger>
<!-- Set level="debug" to see more information about extension initialization -->
<Logger name="org.apache.druid.initialization" level="info" additivity="false">
<Appender-ref ref="FileAppender"/>
</Logger>
<!-- Quieter logging at startup -->
<Logger name="org.skife.config" level="warn" additivity="false">
<Appender-ref ref="FileAppender"/>
</Logger>
<Logger name="com.sun.jersey.guice" level="warn" additivity="false">
<Appender-ref ref="FileAppender"/>
</Logger> </Logger>
-->
</Loggers> </Loggers>
</Configuration> </Configuration>
``` ```
> NOTE: > NOTE:
> Although the log4j configuration file is shared with Druid's task peon processes, > Although Druid shares the log4j configuration file task peon processes,
> the appenders in this file DO NOT take effect for peon processes, which always output logs to standard output. > the appenders in this file DO NOT take effect for peon processes. Peons always output logs to standard output.
> Middle Managers redirect task logs from standard output to [long-term storage](index.md#log-long-term-storage). > Middle Managers redirect task logs from standard output to [long-term storage](index.md#log-long-term-storage).
> >
> However, log level settings do take effect for these task peon processes, > However, log level settings do take effect for these task peon processes.
> which means you can still configure loggers at different logging level for task logs using `log4j2.xml`. > This means you can configure loggers at different logging level for task logs using `log4j2.xml`.
## Log directory ## Log directory
The included log4j2.xml configuration for Druid and ZooKeeper will output logs to the `log` directory at the root of the distribution. The included log4j2.xml configuration for Druid and ZooKeeper writes logs to the `log` directory at the root of the distribution.
If you want to change the log directory, set the environment variable `DRUID_LOG_DIR` to the right directory before you start Druid. If you want to change the log directory, set the environment variable `DRUID_LOG_DIR` to the right directory before you start Druid.
## All-in-one start commands ## All-in-one start commands
If you use one of the all-in-one start commands, such as `bin/start-micro-quickstart`, then in the default configuration If you use one of the all-in-one start commands, such as `bin/start-micro-quickstart`, the default configuration for each service has two kinds of log files.
each service has two kind of log files. The main log file (for example, `log/historical.log`) is written by log4j2 and Log4j2 writes the main log file and rotates it periodically.
is rotated periodically. For example, `log/historical.log`.
The secondary log file (for example, `log/historical.stdout.log`) contains anything that is written by the component The secondary log file contains anything that is written by the component
directly to standard output or standard error without going through log4j2. This consists mainly of messages from the directly to standard output or standard error without going through log4j2.
Java runtime itself. This file is not rotated, but it is generally small due to the low volume of messages. If For example, `log/historical.stdout.log`.
necessary, you can truncate it using the Linux command `truncate --size 0 log/historical.stdout.log`. This consists mainly of messages from the
Java runtime itself.
This file is not rotated, but it is generally small due to the low volume of messages.
If necessary, you can truncate it using the Linux command `truncate --size 0 log/historical.stdout.log`.
## Avoid reflective access warnings in logs ## Avoid reflective access warnings in logs
On Java 11, you may see warnings like this in log files: On Java 11, you may see warnings like the following in the logs:
``` ```
WARNING: An illegal reflective access operation has occurred WARNING: An illegal reflective access operation has occurred
@ -107,9 +145,10 @@ WARNING: All illegal access operations will be denied in a future release
To avoid these, add the `--add-exports` and `--add-opens` command line parameters described in the documentation section To avoid these, add the `--add-exports` and `--add-opens` command line parameters described in the documentation section
about [Java strong encapsulation](../operations/java.md#strong-encapsulation). about [Java strong encapsulation](../operations/java.md#strong-encapsulation).
## My logs are really chatty, can I set them to asynchronously write? ## Set the logs to asynchronously write
Yes, using a `log4j2.xml` similar to the following causes some of the more chatty classes to write asynchronously: If your logs are really chatty, you can set them to write asynchronously.
The following example shows a `log4j2.xml` that configures some of the more chatty classes to write asynchronously:
``` ```
<?xml version="1.0" encoding="UTF-8" ?> <?xml version="1.0" encoding="UTF-8" ?>