Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Quieter streaming supervisors. (#13392) 

Eliminates two common sources of noise with Kafka supervisors that have
large numbers of tasks and partitions:

1) Log the report at DEBUG rather than INFO level at each run cycle.
   It can get quite large, and can be retrieved via API when needed.

2) Use log4j2.xml to quiet down the org.apache.kafka.clients.consumer.internals
   package. Avoids a log message per-partition per-minute as part of seeking
   to the latest offset in the reporting thread. In the tasks, where this
   sort of logging might be more useful, we have another log message with
   the same information: "Seeking partition[%s] to[%s]".
This commit is contained in:
Gian Merlino 2022-11-20 23:53:17 -08:00 committed by GitHub
parent fa3ab27a71
commit c61313f4c4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 6 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
examples/conf/druid/cluster/_common/log4j2.xml
View File

@ -82,5 +82,10 @@
<Logger name="com.sun.jersey.guice" level="warn" additivity="false">
<Appender-ref ref="FileAppender"/>
</Logger>
<!-- Quieter KafkaSupervisors -->
<Logger name="org.apache.kafka.clients.consumer.internals" level="warn" additivity="false">
<Appender-ref ref="FileAppender"/>
</Logger>
</Loggers>
</Configuration>

4
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
View File

@ -1508,9 +1508,7 @@ public abstract class SeekableStreamSupervisor<PartitionIdType, SequenceOffsetTy
private void generateAndLogReport()
{
if (log.isDebugEnabled()) {
log.debug(generateReport(true).toString());
} else {
log.info(generateReport(false).toString());
log.debug("%s", generateReport(true));
}
}