Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Adjust default globalIngestionHeapLimitBytes for indexer, add more docs (#8255)

This commit is contained in:
Jonathan Wei 2019-08-08 01:04:07 -05:00 committed by Fangjin Yang
parent 03a37936ec
commit e88bbe71c0
2 changed files with 9 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/content/development/indexer.md
View File

@ -66,7 +66,13 @@ This global limit is evenly divided across the number of task slots configured b
To apply the per-task heap limit, the Indexer will override `maxBytesInMemory` in task tuning configs (i.e., ignoring the default value or any user configured value). `maxRowsInMemory` will also be overridden to an essentially unlimited value: the Indexer does not support row limits. To apply the per-task heap limit, the Indexer will override `maxBytesInMemory` in task tuning configs (i.e., ignoring the default value or any user configured value). `maxRowsInMemory` will also be overridden to an essentially unlimited value: the Indexer does not support row limits.
By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the available JVM heap. The remaining portion of the heap is reserved for query processing and segment persist/merge operations, and miscellaneous heap usage. By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 1/6th of the available JVM heap. This default is chosen to align with the default value of `maxBytesInMemory` in task tuning configs when using the MiddleManager/Peon system, which is also 1/6th of the JVM heap.
The peak usage for rows held in heap memory relates to the interaction between the `maxBytesInMemory` and `maxPendingPersists` properties in the task tuning configs. When the amount of row data held in-heap by a task reaches the limit specified by `maxBytesInMemory`, a task will persist the in-heap row data. After the persist has been started, the task can again ingest up to `maxBytesInMemory` bytes worth of row data while the persist is running.
This means that the peak in-heap usage for row data can be up to approximately `maxBytesInMemory` * (2 + `maxPendingPersists`). The default value of `maxPendingPersists` is 0, which allows for 1 persist to run concurrently with ingestion work.
The remaining portion of the heap is reserved for query processing and segment persist/merge operations, and miscellaneous heap usage.
#### Concurrent Segment Persist/Merge Limits #### Concurrent Segment Persist/Merge Limits

2
server/src/main/java/org/apache/druid/indexing/worker/config/WorkerConfig.java
View File

@ -50,7 +50,7 @@ public class WorkerConfig
private Period intermediaryPartitionTimeout = new Period("P1D"); private Period intermediaryPartitionTimeout = new Period("P1D");
@JsonProperty @JsonProperty
private final long globalIngestionHeapLimitBytes = (long) (Runtime.getRuntime().maxMemory() * 0.6); private final long globalIngestionHeapLimitBytes = (long) (Runtime.getRuntime().maxMemory() / 6);
@JsonProperty @JsonProperty
private final int numConcurrentMerges = (int) Math.max(1, capacity / 2); private final int numConcurrentMerges = (int) Math.max(1, capacity / 2);