From d50890e92541233cbdedf6d557e5b8b4554660ca Mon Sep 17 00:00:00 2001 From: Shalin Shekhar Mangar Date: Thu, 15 Mar 2018 23:06:52 +0530 Subject: [PATCH] SOLR-12098: Document the Lucene spins auto-detection and its effect on CMS dynamic defaults --- solr/CHANGES.txt | 3 +++ .../src/indexconfig-in-solrconfig.adoc | 20 +++++++++++++++++++ .../src/taking-solr-to-production.adoc | 14 +++++++++++++ 3 files changed, 37 insertions(+) diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt index c2bdea45674..add2014f8b8 100644 --- a/solr/CHANGES.txt +++ b/solr/CHANGES.txt @@ -449,6 +449,9 @@ Other Changes * SOLR-12099: Remove reopenReaders attribute from 'IndexConfig in SolrConfig' page in ref guide. (shalin) +* SOLR-12098: Document the Lucene spins auto-detection and its effect on CMS dynamic defaults. + (Cassandra Targett, shalin) + ================== 7.2.1 ================== Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release. diff --git a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc index e80eb079252..c0c16969bd8 100644 --- a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc +++ b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc @@ -112,11 +112,31 @@ The example above shows Solr's {solr-javadocs}/solr-core/org/apache/solr/index/S The merge scheduler controls how merges are performed. The default `ConcurrentMergeScheduler` performs merges in the background using separate threads. The alternative, `SerialMergeScheduler`, does not perform merges with separate threads. +The `ConcurrentMergeScheduler` has two configurable attributes: + +`maxMergeCount`:: +The maximum number of simultaneous merges that are allowed. If a merge is necessary yet we already have this many threads running, the indexing thread will block until a merge thread has completed. Note that Solr will only run the smallest `maxThreadCount` merges at a time. + +`maxThreadCount`:: +The maximum number of simultaneous merge threads that should be running at once. This must be less than `maxMergeCount`. + +The defaults for the above attributes are dynamically set based on whether the underlying disk drive is rotational disk or not. Refer to the <> section for more details. + +.Example: Dynamic defaults [source,xml] ---- ---- +.Example: Explicit defaults +[source,xml] +---- + + 9 + 4 + +---- + === mergedSegmentWarmer When using Solr in for <> a merged segment warmer can be configured to warm the reader on the newly merged segment, before the merge commits. This is not required for near real-time search, but will reduce search latency on opening a new near real-time reader after a merge completes. diff --git a/solr/solr-ref-guide/src/taking-solr-to-production.adoc b/solr/solr-ref-guide/src/taking-solr-to-production.adoc index 88f127cfc83..4ede6b441f3 100644 --- a/solr/solr-ref-guide/src/taking-solr-to-production.adoc +++ b/solr/solr-ref-guide/src/taking-solr-to-production.adoc @@ -163,6 +163,20 @@ If the `status` command is not successful, look for error messages in `/var/solr == Fine-Tune Your Production Setup +=== Dynamic Defaults for ConcurrentMergeScheduler + +The Merge Scheduler is configured in `solrconfig.xml` and defaults to `ConcurrentMergeScheduler`. This scheduler uses multiple threads to merge Lucene segments in the background. + +By default, the `ConcurrentMergeScheduler` auto-detects whether the underlying disk drive is rotational or a SSD and sets defaults for `maxThreadCount` and `maxMergeCount` accordingly. If the disk drive is determined to be rotational then the `maxThreadCount` is set to 1 and `maxMergeCount` is set to 6. Otherwise, `maxThreadCount` is set to 4 or half the number of processors available to the JVM whichever is greater and `maxMergeCount` is set to `maxThreadCount+5`. + +This auto-detection works only on Linux and even then it is not guaranteed to be correct. On all other platforms, the disk is assumed to be rotational. Therefore, if the auto-detection fails or is incorrect then indexing performance can suffer badly due to the wrong defaults. + +The auto-detected value is exposed by the <> with the key `solr.node:CONTAINER.fs.coreRoot.spins`. A value of `true` denotes that the disk is detected to be a rotational or spinning disk. + +It is safer to explicitly set values for `maxThreadCount` and `maxMergeCount` in the <> so that values appropriate to your hardware are used. + +Alternatively, the boolean system property `lucene.cms.override_spins` can be set in the `SOLR_OPTS` variable in the include file to override the auto-detected value. Similarily, the system property `lucene.cms.override_core_count` can be set to the number of CPU cores to override the auto-detected processor count. + === Memory and GC Settings By default, the `bin/solr` script sets the maximum Java heap size to 512M (-Xmx512m), which is fine for getting started with Solr. For production, you’ll want to increase the maximum heap size based on the memory requirements of your search application; values between 10 and 20 gigabytes are not uncommon for production servers. When you need to change the memory settings for your Solr server, use the `SOLR_JAVA_MEM` variable in the include file, such as: