honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Reduce shard inactivity timeout to 5m 

To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget) to other active shards. The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

Closes #11479
This commit is contained in:
Boaz Leskes 2015-06-03 18:47:44 +02:00
parent 90cbf80fc4
commit 26d71fe00e
2 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/reference/indices/flush.asciidoc
View File

@ -50,7 +50,7 @@ POST /_flush
=== Synced Flush === Synced Flush
Elasticsearch tracks the indexing activity of each shard. Shards that have not Elasticsearch tracks the indexing activity of each shard. Shards that have not
received any indexing operations for 30 minutes are automatically marked as inactive. This presents received any indexing operations for 5 minutes are automatically marked as inactive. This presents
an opportunity for Elasticsearch to reduce shard resources and also perform an opportunity for Elasticsearch to reduce shard resources and also perform
a special kind of flush, called `synced flush`. A synced flush performs a normal flush, then adds a special kind of flush, called `synced flush`. A synced flush performs a normal flush, then adds
a generated unique marker (sync_id) to all shards. a generated unique marker (sync_id) to all shards.
@ -117,7 +117,7 @@ which returns something similar to:
=== Synced Flush API === Synced Flush API
The Synced Flush API allows an administrator to initiate a synced flush manually. This can be particularly useful for The Synced Flush API allows an administrator to initiate a synced flush manually. This can be particularly useful for
a planned (rolling) cluster restart where you can stop indexing and don't want to wait the default 30 minutes for a planned (rolling) cluster restart where you can stop indexing and don't want to wait the default 5 minutes for
idle indices to be sync-flushed automatically. idle indices to be sync-flushed automatically.
While handy, there are a couple of caveats for this API: While handy, there are a couple of caveats for this API:

2
src/main/java/org/elasticsearch/indices/memory/IndexingMemoryController.java
View File

@ -115,7 +115,7 @@ public class IndexingMemoryController extends AbstractLifecycleComponent<Indexin
this.minShardTranslogBufferSize = this.settings.getAsBytesSize("indices.memory.min_shard_translog_buffer_size", new ByteSizeValue(2, ByteSizeUnit.KB)); this.minShardTranslogBufferSize = this.settings.getAsBytesSize("indices.memory.min_shard_translog_buffer_size", new ByteSizeValue(2, ByteSizeUnit.KB));
this.maxShardTranslogBufferSize = this.settings.getAsBytesSize("indices.memory.max_shard_translog_buffer_size", new ByteSizeValue(64, ByteSizeUnit.KB)); this.maxShardTranslogBufferSize = this.settings.getAsBytesSize("indices.memory.max_shard_translog_buffer_size", new ByteSizeValue(64, ByteSizeUnit.KB));
this.inactiveTime = this.settings.getAsTime("indices.memory.shard_inactive_time", TimeValue.timeValueMinutes(30)); this.inactiveTime = this.settings.getAsTime("indices.memory.shard_inactive_time", TimeValue.timeValueMinutes(5));
// we need to have this relatively small to move a shard from inactive to active fast (enough) // we need to have this relatively small to move a shard from inactive to active fast (enough)
this.interval = this.settings.getAsTime("indices.memory.interval", TimeValue.timeValueSeconds(30)); this.interval = this.settings.getAsTime("indices.memory.interval", TimeValue.timeValueSeconds(30));