184 lines
9.6 KiB
Plaintext
184 lines
9.6 KiB
Plaintext
[[bootstrap-checks]]
|
|
== Bootstrap Checks
|
|
|
|
Collectively, we have a lot of experience with users suffering
|
|
unexpected issues because they have not configured
|
|
<<important-settings,important settings>>. In previous versions of
|
|
Elasticsearch, misconfiguration of some of these settings were logged
|
|
as warnings. Understandably, users sometimes miss these log messages.
|
|
To ensure that these settings receive the attention that they deserve,
|
|
Elasticsearch has bootstrap checks upon startup.
|
|
|
|
These bootstrap checks inspect a variety of Elasticsearch and system
|
|
settings and compare them to values that are safe for the operation of
|
|
Elasticsearch. If Elasticsearch is in development mode, any bootstrap
|
|
checks that fail appear as warnings in the Elasticsearch log. If
|
|
Elasticsearch is in production mode, any bootstrap checks that fail will
|
|
cause Elasticsearch to refuse to start.
|
|
|
|
There are some bootstrap checks that are always enforced to prevent
|
|
Elasticsearch from running with incompatible settings. These checks are
|
|
documented individually.
|
|
|
|
[float]
|
|
=== Development vs. production mode
|
|
|
|
By default, Elasticsearch binds to `localhost` for <<modules-http,HTTP>>
|
|
and <<modules-transport,transport (internal)>> communication. This is
|
|
fine for downloading and playing with Elasticsearch, and everyday
|
|
development but it's useless for production systems. To form a cluster,
|
|
Elasticsearch instances must be reachable via transport communication so
|
|
they must bind transport to an external interface. Thus, we consider an
|
|
Elaticsearch instance to be in development mode if it does not bind
|
|
transport to an external interface (the default), and is otherwise in
|
|
production mode if it does bind transport to an external interface. Note
|
|
that HTTP can be configured independently of transport via
|
|
<<modules-http,`http.host`>> and <<modules-transport,`transport.host`>>;
|
|
this can be useful for configuring a single instance to be reachable via
|
|
HTTP for testing purposes without triggering production mode.
|
|
|
|
=== Heap size check
|
|
|
|
If a JVM is started with unequal initial and max heap size, it can be
|
|
prone to pauses as the JVM heap is resized during system usage. To avoid
|
|
these resize pauses, it's best to start the JVM with the initial heap
|
|
size equal to the maximum heap size. Additionally, if
|
|
<<bootstrap.memory_lock,`bootstrap.memory_lock`>> is enabled, the JVM will
|
|
lock the initial size of the heap on startup. If the initial heap size
|
|
is not equal to the maximum heap size, after a resize it will not be the
|
|
case that all of the JVM heap is locked in memory. To pass the heap size
|
|
check, you must configure the <<heap-size,heap size>>.
|
|
|
|
|
|
=== File descriptor check
|
|
|
|
File descriptors are a Unix construct for tracking open "files". In Unix
|
|
though, https://en.wikipedia.org/wiki/Everything_is_a_file[everything is
|
|
a file]. For example, "files" could be a physical file, a virtual file
|
|
(e.g., `/proc/loadavg`), or network sockets. Elasticsearch requires
|
|
lots of file descriptors (e.g., every shard is composed of multiple
|
|
segments and other files, plus connections to other nodes, etc.). This
|
|
bootstrap check is enforced on OS X and Linux. To pass the file
|
|
descriptor check, you might have to configure <<file-descriptors,file
|
|
descriptors>>.
|
|
|
|
=== Memory lock check
|
|
|
|
When the JVM does a major garbage collection it touches every page of
|
|
the heap. If any of those pages are swapped out to disk they will have
|
|
to be swapped back in to memory. That causes lots of disk thrashing that
|
|
Elasticsearch would much rather use to service requests. There are
|
|
several ways to configure a system to disallow swapping. One way is by
|
|
requesting the JVM to lock the heap in memory through `mlockall` (Unix)
|
|
or virtual lock (Windows). This is done via the Elasticsearch setting
|
|
<<bootstrap.memory_lock,`bootstrap.memory_lock`>>. However, there are cases
|
|
where this setting can be passed to Elasticsearch but Elasticsearch is
|
|
not able to lock the heap (e.g., if the `elasticsearch` user does not
|
|
have `memlock unlimited`). The memory lock check verifies that *if* the
|
|
`bootstrap.memory_lock` setting is enabled, that the JVM was successfully
|
|
able to lock the heap. To pass the memory lock check, you might have to
|
|
configure <<mlockall,`mlockall`>>.
|
|
|
|
=== Maximum number of threads check
|
|
|
|
Elasticsearch executes requests by breaking the request down into stages
|
|
and handing those stages off to different thread pool executors. There
|
|
are different <<modules-threadpool,thread pool executors>> for a variety
|
|
of tasks within Elasticsearch. Thus, Elasticsearch needs the ability to
|
|
create a lot of threads. The maximum number of threads check ensures
|
|
that the Elasticsearch process has the rights to create enough threads
|
|
under normal use. This check is enforced only on Linux. If you are on
|
|
Linux, to pass the maximum number of threads check, you must configure
|
|
your system to allow the Elasticsearch process the ability to create at
|
|
least 2048 threads. This can be done via `/etc/security/limits.conf`
|
|
using the `nproc` setting (note that you might have to increase the
|
|
limits for the `root` user too).
|
|
|
|
[[max-size-virtual-memory-check]]
|
|
=== Maximum size virtual memory check
|
|
|
|
Elasticsearch and Lucene use `mmap` to great effect to map portions of
|
|
an index into the Elasticsearch address space. This keeps certain index
|
|
data off the JVM heap but in memory for blazing fast access. For this to
|
|
be effective, the Elasticsearch should have unlimited address space. The
|
|
maximum size virtual memory check enforces that the Elasticsearch
|
|
process has unlimited address space and is enforced only on Linux. To
|
|
pass the maximum size virtual memory check, you must configure your
|
|
system to allow the Elasticsearch process the ability to have unlimited
|
|
address space. This can be done via `/etc/security/limits.conf` using
|
|
the `as` setting to `unlimited` (note that you might have to increase
|
|
the limits for the `root` user too).
|
|
|
|
=== Maximum map count check
|
|
|
|
Continuing from the previous <<max-size-virtual-memory-check,point>>, to
|
|
use `mmap` effectively, Elasticsearch also requires the ability to
|
|
create many memory-mapped areas. The maximum map count check checks that
|
|
the kernel allows a process to have at least 262,144 memory-mapped areas
|
|
and is enforced on Linux only. To pass the maximum map count check, you
|
|
must configure `vm.max_map_count` via `sysctl` to be at least `262144`.
|
|
|
|
=== Client JVM check
|
|
|
|
There are two different JVMs provided by OpenJDK-derived JVMs: the
|
|
client JVM and the server JVM. These JVMs use different compilers for
|
|
producing executable machine code from Java bytecode. The client JVM is
|
|
tuned for startup time and memory footprint while the server JVM is
|
|
tuned for maximizing performance. The difference in performance between
|
|
the two VMs can be substantial. The client JVM check ensures that
|
|
Elasticsearch is not running inside the client JVM. To pass the client
|
|
JVM check, you must start Elasticsearch with the server VM. On modern
|
|
systems and operating systems, the server VM is the
|
|
default. Additionally, Elasticsearch is configured by default to force
|
|
the server VM.
|
|
|
|
=== Use serial collector check
|
|
|
|
There are various garbage collectors for the OpenJDK-derived JVMs targeting
|
|
different workloads. The serial collector in particular is best suited for
|
|
single logical CPU machines or extremely small heaps, neither of which are
|
|
suitable for running Elasticsearch. Using the serial collector with
|
|
Elasticsearch can be devastating for performance. The serial collector check
|
|
ensures that Elasticsearch is not configured to run with the serial
|
|
collector. To pass the serial collector check, you must not start Elasticsearch
|
|
with the serial collector (whether it's from the defaults for the JVM that
|
|
you're using, or you've explicitly specified it with `-XX:+UseSerialGC`). Note
|
|
that the default JVM configuration that ship with Elasticsearch configures
|
|
Elasticsearch to use the CMS collector.
|
|
|
|
=== System call filter check
|
|
|
|
Elasticsearch installs system call filters of various flavors depending on the
|
|
operating system (e.g., seccomp on Linux). These system call filters are
|
|
installed to prevent the ability to execute system calls related to forking as
|
|
a defense mechanism against arbitrary code execution attacks on Elasticsearch
|
|
The system call filter check ensures that if system call filters are enabled,
|
|
then they were successfully installed. To pass the system call filter check you
|
|
must either fix any configuration errors on your system that prevented system
|
|
call filters from installing (check your logs), or *at your own risk* disable
|
|
system call filters by setting `bootstrap.seccomp` to `false`.
|
|
|
|
=== OnError and OnOutOfMemoryError checks
|
|
|
|
The JVM options `OnError` and `OnOutOfMemoryError` enable executing
|
|
arbitrary commands if the JVM encounters a fatal error (`OnError`) or an
|
|
`OutOfMemoryError` (`OnOutOfMemoryError`). However, by default,
|
|
Elasticsearch system call filters (seccomp) are enabled and these
|
|
filters prevent forking. Thus, using `OnError` or `OnOutOfMemoryError`
|
|
and system call filters are incompatible. The `OnError` and
|
|
`OnOutOfMemoryError` checks prevent Elasticsearch from starting if
|
|
either of these JVM options are used and system call filters are
|
|
enabled. This check is always enforced. To pass this check do not enable
|
|
`OnError` nor `OnOutOfMemoryError`; instead, upgrade to Java 8u92 and
|
|
use the JVM flag `ExitOnOutOfMemoryError`. While this does not have the
|
|
full capabilities of `OnError` nor `OnOutOfMemoryError`, arbitrary
|
|
forking will not be supported with seccomp enabled.
|
|
|
|
=== G1GC check
|
|
|
|
Early versions of the HotSpot JVM that shipped with JDK 8 are known to have
|
|
issues that can lead to index corruption when the G1GC collector is enabled.
|
|
The versions impacted are those earlier than the version of HotSpot that
|
|
shipped with JDK 8u40. The G1GC check detects these early versions of the
|
|
HotSpot JVM.
|