From c41e63c2f952f593d5d29853c9aa46b30e044206 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 12 Jun 2014 13:56:06 +0200 Subject: [PATCH] Docs: Updated index-modules/store and setup/configuration Explain how to set different index storage types, and added the vm settings required to stop mmapfs from running out of memory Closes #6327 --- docs/reference/index-modules/store.asciidoc | 34 +++++++++++++++++---- docs/reference/setup/configuration.asciidoc | 16 ++++++++++ 2 files changed, 44 insertions(+), 6 deletions(-) diff --git a/docs/reference/index-modules/store.asciidoc b/docs/reference/index-modules/store.asciidoc index 875c06a9a19..6fae6ec013a 100644 --- a/docs/reference/index-modules/store.asciidoc +++ b/docs/reference/index-modules/store.asciidoc @@ -50,21 +50,40 @@ is `node`, meaning it will throttle based on the node level settings and participate in the global throttling happening. Both settings can be set using the index update settings API dynamically. -The following sections lists all the different storage types supported. - [float] [[file-system]] -=== File System +=== File system storage types File system based storage is the default storage used. There are -different implementations or storage types. The best one for the +different implementations or _storage types_. The best one for the operating environment will be automatically chosen: `mmapfs` on Solaris/Linux/Windows 64bit, `simplefs` on Windows 32bit, and `niofs` for the rest. -The following are the different file system based storage types: +This can be overridden for all indices by adding this to the +`config/elasticsearch.yml` file: + +[source,yaml] +--------------------------------- +index.store.type: niofs +--------------------------------- + +It can also be set on a per-index basis at index creation time: + +[source,json] +--------------------------------- +curl -XPUT localhost:9200/my_index +{ + "settings": { + "index.store.type": "niofs" + } +} +--------------------------------- + +The following sections lists all the different storage types supported. [float] +[[simplefs]] ==== Simple FS The `simplefs` type is a straightforward implementation of file system @@ -74,6 +93,7 @@ will bottleneck). It is usually better to use the `niofs` when you need index persistence. [float] +[[niofs]] ==== NIO FS The `niofs` type stores the shard index on the file system (maps to @@ -90,9 +110,11 @@ Lucene `MMapDirectory`) by mapping a file into memory (mmap). Memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space. +See <> [float] [[store-memory]] === Memory -The `memory` type stores the index in main memory. +The `memory` type stores the index in main memory, using Lucene's +`RamIndexStore`. diff --git a/docs/reference/setup/configuration.asciidoc b/docs/reference/setup/configuration.asciidoc index 1ba100ddaf9..18bfd468558 100644 --- a/docs/reference/setup/configuration.asciidoc +++ b/docs/reference/setup/configuration.asciidoc @@ -47,6 +47,22 @@ using the <> API, with: curl localhost:9200/_nodes/process?pretty -------------------------------------------------- +[float] +[[vm-max-map-count]] +==== Virtual memory + +Elasticsearch uses <> by default to store its indices. The default +operating system limits on mmap counts is likely to be too low, which may +result in out of memory exceptions. On Linux, you can increase the limits by +running the following command as `root`: + +[source,bash] +------------------------------------- +sysctl -w vm.max_map_count=262144 +------------------------------------- + +To set this value permanently, update the `vm.max_map_count` setting in +`/etc/sysctl.conf`. [float] [[setup-configuration-memory]]