SOLR-10306: Document in Reference Guide how to disable or reduce swapping (#1256)

solr/CHANGES.txt

@@ -200,6 +200,8 @@ Other Changes
 * SOLR-11035: (at least) 2 distinct failures possible when clients attempt searches
   during SolrCore reload. Removed Bandaid kludge. (Erick Erickson)
 
+* SOLR-10306: Document in Reference Guide how to disable or reduce swapping (janhoy)
+
 ==================  8.4.1 ==================
 
 Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.

solr/solr-ref-guide/src/jvm-settings.adoc

@@ -26,7 +26,7 @@ For more general information about improving Solr performance, see https://cwiki
 
 The most important JVM configuration settings control the heap allocated to the JVM: `-Xms`, which sets the initial size of the JVM's memory heap, and `-Xmx`, which sets the maximum size of the heap. Setting these two options to the same value is a common practice.
 
-Heap size is critical and unfortunately there is no "one size fits all" solution, you must test with your data and your application. The best way to determine the correct size is to analyze the garbage collection (GC) logs located in your logs directory. There are various tools that help analyze these logs and, in particular, show the amount of memory used after GC has completed (GCViewer and GCEasy are two). Also you can attach jconsole (distributed with most Java runtimes) to check memory consumption as Solr is running. This will show the absolute miminum amount of memory required; adding 25-50% "headroom" is a reasonable starting point.
+Heap size is critical and unfortunately there is no "one size fits all" solution, you must test with your data and your application. The best way to determine the correct size is to analyze the garbage collection (GC) logs located in your logs directory. There are various tools that help analyze these logs and, in particular, show the amount of memory used after GC has completed (GCViewer and GCEasy are two). Also you can attach jconsole (distributed with most Java runtimes) to check memory consumption as Solr is running. This will show the absolute minimum amount of memory required; adding 25-50% "headroom" is a reasonable starting point.
 
 There are several points to keep in mind:
 
@@ -34,7 +34,7 @@ There are several points to keep in mind:
  * Lucene/Solr makes extensive use of MMapDirectory, which uses RAM _not_ reserved for the JVM for most of the Lucene index. Therefore, as much memory as possible should be left for the operating system to use for this purpose.
  * The heap allocated should be as small as possible while maintaining good performance. 8-16G is quite common, and larger heaps are sometimes used. When heaps grow to larger sizes, it is imperative to test extensively before going to production.
  * The G1GC garbage collector is currently preferred when using a JVM that supports it (Java 9 and later)
- * Modern hardware can be configured with hundreds of gigabytes of physical RAM and many CPUs. It is often better in these cases to run multiple JVMs, each with a limited amount of memory allocated to their heaps.
+ * Modern hardware can be configured with hundreds of gigabytes of physical RAM and many CPUs. It is often better in these cases to run multiple JVMs, each with a limited amount of memory allocated to their heaps. One way to achieve this is to run Solr as a Docker container.
  * It's good practice to periodically re-analyze the GC logs and/or monitor with <<metrics-reporting#metrics-reporting,Metrics Reporting>> to see if the memory usage has changed due to changes in your application, number of documents, etc.
  * On *nix systems, we recommend that Solr be run with the "oom killer script" (see solr/bin/oom_solr.sh). This will forcefully stop Solr when the heap is exhausted rather than continue in an indeterminate state.
  * All current (Java 11) garbage collectors can hit "stop the world" collections, which suspend the JVM until completed. If, through monitoring, these collections are frequent and greater than your application can tolerate, additional tuning should be considered. "Stop the world" pauses greater than 5 seconds are rarely acceptable, and having them be less than 1 second is desirable.
@@ -47,4 +47,4 @@ If you are using Sun's JVM, add the `-server` command-line option when you start
 
 == Checking JVM Settings
 
-A great way to see what JVM settings your server is using, along with other useful information, is to use the admin RequestHandler, `solr/admin/system`. This request handler will display a wealth of server statistics and settings.
+A great way to see what JVM settings your server is using, along with other useful information, is to use the admin RequestHandler, `/solr/admin/info/system`. This request handler will display a wealth of server statistics and settings.

solr/solr-ref-guide/src/taking-solr-to-production.adoc

@@ -179,13 +179,18 @@ Alternatively, the boolean system property `lucene.cms.override_spins` can be se
 
 === Memory and GC Settings
 
-By default, the `bin/solr` script sets the maximum Java heap size to 512M (-Xmx512m), which is fine for getting started with Solr. For production, you’ll want to increase the maximum heap size based on the memory requirements of your search application; values between 10 and 20 gigabytes are not uncommon for production servers. When you need to change the memory settings for your Solr server, use the `SOLR_JAVA_MEM` variable in the include file, such as:
+By default, the `bin/solr` script sets the maximum Java heap size to 512M (-Xmx512m), which is fine for getting started with Solr. For production, you’ll want to increase the maximum heap size based on the memory requirements of your search application; values between 8 and 16 gigabytes are not uncommon for production servers. When you need to change the memory settings for your Solr server, use the `SOLR_HEAP` variable in the include file, such as:
 
 [source,bash]
 ----
-SOLR_JAVA_MEM="-Xms10g -Xmx10g"
+SOLR_HEAP="8g"
 ----
 
+[NOTE]
+====
+Do not allocate a very large Java Heap unless you know you need it. See <<jvm-settings.adoc#choosing-memory-heap-settings,Choosing Memory Heap Settings>> for details.
+====
+
 Also, the <<solr-control-script-reference.adoc#solr-control-script-reference,Solr Control Script>> comes with a set of pre-configured Garbage First Garbage Collection settings that have shown to work well with Solr for a number of different workloads.
 However, these settings may not work well for your specific use of Solr. Consequently, you may need to change the GC settings, which should also be done with the `GC_TUNE` variable in the `/etc/default/solr.in.sh` include file. For more information about garbage collection settings refer to following articles:
 1. https://cwiki.apache.org/confluence/display/solr/ShawnHeisey
@@ -239,7 +244,7 @@ SOLR_HOST=solr1.example.com
 
 Setting the hostname of the Solr server is recommended, especially when running in SolrCloud mode, as this determines the address of the node when it registers with ZooKeeper.
 
-=== Environment banner in Admin UI
+=== Environment Banner in Admin UI
 
 To guard against accidentally doing changes to the wrong cluster, you may configure a visual indication in the Admin UI of whether you currently work with a production environment or not. To do this, edit your `solr.in.sh` or `solr.in.cmd` file with a `-Dsolr.environment=prod` setting, or set the cluster property named `environment`. To specify label and/or color, use a comma delimited format as below. The `+` character can be used instead of space to avoid quoting. Colors may be valid CSS colors or numeric, e.g., `#ff0000` for bright red. Examples of valid environment configs:
 
@@ -302,6 +307,44 @@ Check these limits every time you upgrade your kernel or operating system. These
 If these limits are exceeded, the problems reported by Solr vary depending on the specific operation responsible for exceeding the limit. Errors such as "too many open files", "connection error", and "max processes exceeded" have been reported, as well as SolrCloud recovery failures.
 ====
 
+=== Avoid Swapping (*nix Operating Systems)
+
+When running a Java application like Lucene/Solr, having the OS swap memory to disk is a very bad situation. We usually prefer a hard crash so other healthy Solr nodes can take over, instead of letting a Solr node swap, causing terrible performance, timeouts and an unstable system. So our recommendation is to disable swap on the host altogether or reduce the "swappiness". These instructions are valid for Linux environments. Also note that when running Solr in a Docker container, these changes must be applied to the *host*.
+
+==== Disabling Swap
+
+To disable swap on a Linux system temporarily, you can run the `swapoff` command:
+
+[source,bash]
+----
+sudo swapoff -a
+----
+
+If you want to make this setting permanent, first make sure you have more than enough physical RAM on your host and then consult the documentation for your Linux system for the correct procedure to disable swap.
+
+==== Reduce Swappiness
+
+An alternative option is to reduce the "swappiness" of your system. A Linux system will by default be quite aggressive in swapping out memory to disk, by having a high default "swappiness" value. By reducing this to a very low value, it will have almost the same effect as using `swapoff`, but Linux will still be allowed to swap in case of emergency. To check the current swappiness setting, run:
+
+[source,bash]
+----
+cat /proc/sys/vm/swappiness
+----
+
+Next, to change the setting permanently, open `/etc/sysctl.conf` as the root user. Then, change or add this line to the file:
+
+[source,bash]
+----
+vm.swappiness = 1
+----
+
+Alternatively, you can change the setting temporarily by running this comamnd:
+
+[source,bash]
+----
+echo 1 > /proc/sys/vm/swappiness
+----
+
 == Security Considerations
 
 include::securing-solr.adoc[tag=security-network-binding-1]