Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Make lookup offheap buffer configurable (#5696) 

* Make lookup offheap buffer configurable

Fixes #3663

* Address comments

* Update docs

* Update docs
This commit is contained in:
Dylan Wylie 2018-05-04 18:00:55 +01:00 committed by Slim Bouguerra
parent c2b5e5ec95
commit 2c5f0038fd
3 changed files with 21 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/content/development/extensions-core/lookups-cached-global.md
View File

@ -147,13 +147,16 @@ setting namespaces (broker, peon, historical)
|--------|-----------|-------|
|`druid.lookup.namespace.cache.type`|Specifies the type of caching to be used by the namespaces. May be one of [`offHeap`, `onHeap`]. `offHeap` uses a temporary file for off-heap storage of the namespace (memory mapped files). `onHeap` stores all cache on the heap in standard java map types.|`onHeap`|
|`druid.lookup.namespace.numExtractionThreads`|The number of threads in the thread pool dedicated for lookup extraction and updates. This number may need to be scaled up, if you have a lot of lookups and they take long time to extract, to avoid timeouts.|2|
|`druid.lookup.namespace.numBufferedEntries`|If using offHeap caching, the number of records to be stored on an on-heap buffer.|100,000|
The cache is populated in different ways depending on the settings below. In general, most namespaces employ
a `pollPeriod` at the end of which time they poll the remote resource of interest for updates.
`onHeap` uses `ConcurrentMap`s in the java heap, and thus affects garbage collection and heap sizing.
`offHeap` uses a 10MB on-heap buffer and MapDB using memory-mapped files in the java temporary directory.
So if total `cachedNamespace` lookup size is in excess of 10MB, the extra will be kept in memory as page cache, and paged in and out by general OS tunings.
`offHeap` uses an on-heap buffer and MapDB using memory-mapped files in the java temporary directory.
So if total number of entries in the `cachedNamespace` is in excess of the buffer's configured capacity, the extra will be kept in memory as page cache, and paged in and out by general OS tunings.
It's highly recommended that `druid.lookup.namespace.numBufferedEntries` is set when using `offHeap`, the value should be chosen from the range between 10% and 50% of the number of entries in the lookup.
# Supported Lookups

15
extensions-core/lookups-cached-global/src/main/java/io/druid/server/lookup/namespace/NamespaceExtractionConfig.java
View File

@ -31,6 +31,9 @@ public class NamespaceExtractionConfig
@JsonProperty
private int numExtractionThreads = 2;
@JsonProperty
private int numBufferedEntries = 100_000;
public int getNumExtractionThreads()
{
return numExtractionThreads;
@ -40,4 +43,16 @@ public class NamespaceExtractionConfig
{
this.numExtractionThreads = numExtractionThreads;
}
public int getNumBufferedEntries()
{
return numBufferedEntries;
}
public void setNumBufferedEntries(int numBufferedEntries)
{
this.numBufferedEntries = numBufferedEntries;
}
}

2
extensions-core/lookups-cached-global/src/main/java/io/druid/server/lookup/namespace/cache/OffHeapNamespaceExtractionCacheManager.java vendored
View File

@ -156,7 +156,7 @@ public class OffHeapNamespaceExtractionCacheManager extends NamespaceExtractionC
.asyncWriteEnable()
.mmapFileEnable()
.commitFileSyncDisable()
.cacheSize(10_000_000)
.cacheSize(config.getNumBufferedEntries())
.make();
try {
lifecycle.addMaybeStartHandler(