honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/docs/reference/index-modules/slowlog.asciidoc

119 lines
5.7 KiB
Plaintext
Raw Normal View History

Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
[[index-modules-slowlog]]
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
== Slow Log
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
[float]
Add more anchor links to documentation Related to #3679
2013-09-25 12:17:40 -04:00
[[search-slow-log]]
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
=== Search Slow Log
Shard level slow search log allows to log slow search (query and fetch
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
phases) into a dedicated log file.
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
Thresholds can be set for both the query phase of the execution, and
fetch phase, here is a sample:
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
[source,yaml]
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
--------------------------------------------------
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms
index.search.slowlog.threshold.fetch.warn: 1s
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms
Add search slowlog level to docs (#29040) This commit adds an indication how to set the search slowlog level to the docs.
2018-03-13 18:27:14 -04:00
index.search.slowlog.level: info
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
--------------------------------------------------
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
All of the above settings are _dynamic_ and are set per-index.
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
By default, none are enabled (set to `-1`). Levels (`warn`, `info`,
`debug`, `trace`) allow to control under which logging level the log
will be logged. Not all are required to be configured (for example, only
`warn` threshold can be set). The benefit of several levels is the
ability to quickly "grep" for specific thresholds breached.
The logging is done on the shard level scope, meaning the execution of a
search request within a specific shard. It does not encompass the whole
search request, which can be broadcast to several shards in order to
execute. Some of the benefits of shard level logging is the association
of the actual execution on the specific machine, compared with request
level.
The logging file is configured by default using the following
Update docs for Log4j 2 This commit updates the logging docs for Elasticsearch to reflect the migration to Log4j 2.
2016-08-31 15:51:52 -04:00
configuration (found in `log4j2.properties`):
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
[source,properties]
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
--------------------------------------------------
Update docs for Log4j 2 This commit updates the logging docs for Elasticsearch to reflect the migration to Log4j 2.
2016-08-31 15:51:52 -04:00
appender.index_search_slowlog_rolling.type = RollingFile
appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
Logging: Unify log rotation for index/search slow log (#27298) The existing log rotation configuration allowed the index and search slow log to grow unbounded. This commit removes the date based rotation and adds the same size based rotation, that the depreciation log already has.
2017-11-15 04:01:32 -05:00
appender.index_search_slowlog_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_search_slowlog.log
Update docs for Log4j 2 This commit updates the logging docs for Elasticsearch to reflect the migration to Log4j 2.
2016-08-31 15:51:52 -04:00
appender.index_search_slowlog_rolling.layout.type = PatternLayout
Logging: Make node name consistent in logger (#31588) First, some background: we have 15 different methods to get a logger in Elasticsearch but they can be broken down into three broad categories based on what information is provided when building the logger. Just a class like: ``` private static final Logger logger = ESLoggerFactory.getLogger(ActionModule.class); ``` or: ``` protected final Logger logger = Loggers.getLogger(getClass()); ``` The class and settings: ``` this.logger = Loggers.getLogger(getClass(), settings); ``` Or more information like: ``` Loggers.getLogger("index.store.deletes", settings, shardId) ``` The goal of the "class and settings" variant is to attach the node name to the logger. Because we don't always have the settings available, we often use the "just a class" variant and get loggers without node names attached. There isn't any real consistency here. Some loggers get the node name because it is convenient and some do not. This change makes the node name available to all loggers all the time. Almost. There are some caveats are testing that I'll get to. But in *production* code the node name is node available to all loggers. This means we can stop using the "class and settings" variants to fetch loggers which was the real goal here, but a pleasant side effect is that the ndoe name is now consitent on every log line and optional by editing the logging pattern. This is all powered by setting the node name statically on a logging formatter very early in initialization. Now to tests: tests can't set the node name statically because subclasses of `ESIntegTestCase` run many nodes in the same jvm, even in the same class loader. Also, lots of tests don't run with a real node so they don't *have* a node name at all. To support multiple nodes in the same JVM tests suss out the node name from the thread name which works surprisingly well and easy to test in a nice way. For those threads that are not part of an `ESIntegTestCase` node we stick whatever useful information we can get form the thread name in the place of the node name. This allows us to keep the logger format consistent.
2018-07-31 10:54:24 -04:00
appender.index_search_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker %.-10000m%n
Logging: Unify log rotation for index/search slow log (#27298) The existing log rotation configuration allowed the index and search slow log to grow unbounded. This commit removes the date based rotation and adds the same size based rotation, that the depreciation log already has.
2017-11-15 04:01:32 -05:00
appender.index_search_slowlog_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_search_slowlog-%i.log.gz
Update docs for Log4j 2 This commit updates the logging docs for Elasticsearch to reflect the migration to Log4j 2.
2016-08-31 15:51:52 -04:00
appender.index_search_slowlog_rolling.policies.type = Policies
Logging: Unify log rotation for index/search slow log (#27298) The existing log rotation configuration allowed the index and search slow log to grow unbounded. This commit removes the date based rotation and adds the same size based rotation, that the depreciation log already has.
2017-11-15 04:01:32 -05:00
appender.index_search_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.index_search_slowlog_rolling.policies.size.size = 1GB
appender.index_search_slowlog_rolling.strategy.type = DefaultRolloverStrategy
appender.index_search_slowlog_rolling.strategy.max = 4
Update docs for Log4j 2 This commit updates the logging docs for Elasticsearch to reflect the migration to Log4j 2.
2016-08-31 15:51:52 -04:00
logger.index_search_slowlog_rolling.name = index.search.slowlog
logger.index_search_slowlog_rolling.level = trace
logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
logger.index_search_slowlog_rolling.additivity = false
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
--------------------------------------------------
[float]
Add more anchor links to documentation Related to #3679
2013-09-25 12:17:40 -04:00
[[index-slow-log]]
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
=== Index Slow log
Remove garbage
2014-04-15 00:55:39 -04:00
The indexing slow log, similar in functionality to the search slow
file is -> file name (#18994)
2016-06-21 07:20:23 -04:00
log. The log file name ends with `_index_indexing_slowlog.log`. Log and
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
the thresholds are configured in the same way as the search slowlog.
Index slowlog sample:
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
[source,yaml]
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
--------------------------------------------------
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms
Logging: Log less source in slowlog Instead of logging the entire `_source` in the indexing slowlog we log by default just the first 1000 characters - this is controlled by the `index.indexing.slowlog.source` settings and can be set to `true` to log the whole `_source`, `false` to log none of it, and a number to log at most that many characters. Closes #4485
2015-08-11 16:13:43 -04:00
index.indexing.slowlog.level: info
index.indexing.slowlog.source: 1000
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
--------------------------------------------------
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
All of the above settings are _dynamic_ and are set per-index.
Docs: Refactored modules and index modules sections
2015-06-22 17:49:45 -04:00
Logging: Log less source in slowlog Instead of logging the entire `_source` in the indexing slowlog we log by default just the first 1000 characters - this is controlled by the `index.indexing.slowlog.source` settings and can be set to `true` to log the whole `_source`, `false` to log none of it, and a number to log at most that many characters. Closes #4485
2015-08-11 16:13:43 -04:00
By default Elasticsearch will log the first 1000 characters of the _source in
the slowlog. You can change that with `index.indexing.slowlog.source`. Setting
it to `false` or `0` will skip logging the source entirely an setting it to
Restores the original default format of search slow log In 5.0, the search slow log switched to the multi-line format with no option to get back to the origin single-line format that was used prior to 5.0 by default. This commit removes the reformat option from the search slow log and returns the search slow log back to the single-line format. Closes #21711
2016-11-28 13:12:43 -05:00
`true` will log the entire source regardless of size. The original `_source` is
reformatted by default to make sure that it fits on a single log line. If preserving
the original document format is important, you can turn off reformatting by setting
`index.indexing.slowlog.reformat` to `false`, which will cause the source to be
logged "as is" and can potentially span multiple log lines.
Logging: Log less source in slowlog Instead of logging the entire `_source` in the indexing slowlog we log by default just the first 1000 characters - this is controlled by the `index.indexing.slowlog.source` settings and can be set to `true` to log the whole `_source`, `false` to log none of it, and a number to log at most that many characters. Closes #4485
2015-08-11 16:13:43 -04:00
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
The index slow log file is configured by default in the `log4j2.properties`
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
file:
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
[source,properties]
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
--------------------------------------------------
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
appender.index_indexing_slowlog_rolling.type = RollingFile
appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
Logging: Unify log rotation for index/search slow log (#27298) The existing log rotation configuration allowed the index and search slow log to grow unbounded. This commit removes the date based rotation and adds the same size based rotation, that the depreciation log already has.
2017-11-15 04:01:32 -05:00
appender.index_indexing_slowlog_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_indexing_slowlog.log
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
appender.index_indexing_slowlog_rolling.layout.type = PatternLayout
Logging: Make node name consistent in logger (#31588) First, some background: we have 15 different methods to get a logger in Elasticsearch but they can be broken down into three broad categories based on what information is provided when building the logger. Just a class like: ``` private static final Logger logger = ESLoggerFactory.getLogger(ActionModule.class); ``` or: ``` protected final Logger logger = Loggers.getLogger(getClass()); ``` The class and settings: ``` this.logger = Loggers.getLogger(getClass(), settings); ``` Or more information like: ``` Loggers.getLogger("index.store.deletes", settings, shardId) ``` The goal of the "class and settings" variant is to attach the node name to the logger. Because we don't always have the settings available, we often use the "just a class" variant and get loggers without node names attached. There isn't any real consistency here. Some loggers get the node name because it is convenient and some do not. This change makes the node name available to all loggers all the time. Almost. There are some caveats are testing that I'll get to. But in *production* code the node name is node available to all loggers. This means we can stop using the "class and settings" variants to fetch loggers which was the real goal here, but a pleasant side effect is that the ndoe name is now consitent on every log line and optional by editing the logging pattern. This is all powered by setting the node name statically on a logging formatter very early in initialization. Now to tests: tests can't set the node name statically because subclasses of `ESIntegTestCase` run many nodes in the same jvm, even in the same class loader. Also, lots of tests don't run with a real node so they don't *have* a node name at all. To support multiple nodes in the same JVM tests suss out the node name from the thread name which works surprisingly well and easy to test in a nice way. For those threads that are not part of an `ESIntegTestCase` node we stick whatever useful information we can get form the thread name in the place of the node name. This allows us to keep the logger format consistent.
2018-07-31 10:54:24 -04:00
appender.index_indexing_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker %.-10000m%n
Logging: Unify log rotation for index/search slow log (#27298) The existing log rotation configuration allowed the index and search slow log to grow unbounded. This commit removes the date based rotation and adds the same size based rotation, that the depreciation log already has.
2017-11-15 04:01:32 -05:00
appender.index_indexing_slowlog_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_indexing_slowlog-%i.log.gz
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
appender.index_indexing_slowlog_rolling.policies.type = Policies
Logging: Unify log rotation for index/search slow log (#27298) The existing log rotation configuration allowed the index and search slow log to grow unbounded. This commit removes the date based rotation and adds the same size based rotation, that the depreciation log already has.
2017-11-15 04:01:32 -05:00
appender.index_indexing_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.index_indexing_slowlog_rolling.policies.size.size = 1GB
appender.index_indexing_slowlog_rolling.strategy.type = DefaultRolloverStrategy
appender.index_indexing_slowlog_rolling.strategy.max = 4
Fix slowlog docs This commit fixes two issues with the slow log docs: - clarifies that these settings are per index - updates index slow log configuration for Log4j 2 Relates #20976
2016-10-17 10:50:32 -04:00
logger.index_indexing_slowlog.name = index.indexing.slowlog.index
logger.index_indexing_slowlog.level = trace
logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
logger.index_indexing_slowlog.additivity = false
Migrated documentation into the main repo
2013-08-28 19:24:34 -04:00
--------------------------------------------------