mirror of https://github.com/apache/druid.git
Move caffeine out of extension and make it the default cache implementation. (#4810)
* Move caffeine out of extension. * Remove `JsonTypeName` from the class itself * Fix bad docs * Fix distribution pom * Fix unused import * Make caffeine default * Address code comments * Add more description around the jre version in the readme * Add suggested comments
This commit is contained in:
Charles Allen
GitHub
parent
09fcb75583
commit
a6470c1d03
|
|
@ -79,8 +79,6 @@
|
|||
<argument>-c</argument>
|
||||
<argument>io.druid.extensions:druid-avro-extensions</argument>
|
||||
<argument>-c</argument>
|
||||
<argument>io.druid.extensions:druid-caffeine-cache</argument>
|
||||
<argument>-c</argument>
|
||||
<argument>io.druid.extensions:druid-datasketches</argument>
|
||||
<argument>-c</argument>
|
||||
<argument>io.druid.extensions:druid-hdfs-storage</argument>
|
||||
|
|
|
|||
|
|
@ -113,5 +113,6 @@
|
|||
{"source": "tutorials/tutorial-all-about-queries.html", "target": "quickstart.html"},
|
||||
{"source": "tutorials/tutorial-loading-batch-data.html", "target": "tutorial-batch.html"},
|
||||
{"source": "tutorials/tutorial-loading-streaming-data.html", "target": "tutorial-streams.html"},
|
||||
{"source": "tutorials/tutorial-the-druid-cluster.html", "target": "cluster.html"}
|
||||
{"source": "tutorials/tutorial-the-druid-cluster.html", "target": "cluster.html"},
|
||||
{"source": "development/extensions-core/caffeine-cache.html", "target":"../../configuration/caching.html"}
|
||||
]
|
||||
|
|
|
|||
|
|
@ -24,6 +24,12 @@ for both broker and historical nodes, when defined in the common properties file
|
|||
|
||||
#### Local Cache
|
||||
|
||||
<div class="note caution">
|
||||
DEPRECATED: Use caffeine instead
|
||||
</div>
|
||||
|
||||
The local cache is deprecated in favor of the Caffeine cache, and may be removed in a future version of Druid. The Caffeine cache affords significantly better performance and control over eviction behavior compared to `local` cache, and is recommended in any situation where you are using JRE 8u60 or higher.
|
||||
|
||||
A simple in-memory LRU cache. Local cache resides in JVM heap memory, so if you enable it, make sure you increase heap size accordingly.
|
||||
|
||||
|Property|Description|Default|
|
||||
|
|
@ -32,6 +38,39 @@ A simple in-memory LRU cache. Local cache resides in JVM heap memory, so if you
|
|||
|`druid.cache.initialSize`|Initial size of the hashtable backing the cache.|500000|
|
||||
|`druid.cache.logEvictionCount`|If non-zero, log cache eviction every `logEvictionCount` items.|0|
|
||||
|
||||
### Caffeine Cache
|
||||
|
||||
A highly performant local cache implementation for Druid based on [Caffeine](https://github.com/ben-manes/caffeine). Requires a JRE8u60 or higher if using `COMMON_FJP`.
|
||||
|
||||
##### Configuration
|
||||
|
||||
Below are the configuration options known to this module:
|
||||
|
||||
|`runtime.properties`|Description|Default|
|
||||
|--------------------|-----------|-------|
|
||||
|`druid.cache.type`| Set this to `caffeine`|`local`|
|
||||
|`druid.cache.sizeInBytes`|The maximum size of the cache in bytes on heap.|None (unlimited)|
|
||||
|`druid.cache.expireAfter`|The time (in ms) after an access for which a cache entry may be expired|None (no time limit)|
|
||||
|`druid.cache.cacheExecutorFactory`|The executor factory to use for Caffeine maintenance. One of `COMMON_FJP`, `SINGLE_THREAD`, or `SAME_THREAD`|ForkJoinPool common pool (`COMMON_FJP`)|
|
||||
|`druid.cache.evictOnClose`|If a close of a namespace (ex: removing a segment from a node) should cause an eager eviction of associated cache values|`false`|
|
||||
|
||||
##### `druid.cache.cacheExecutorFactory`
|
||||
|
||||
Here are the possible values for `druid.cache.cacheExecutorFactory`, which controls how maintenance tasks are run
|
||||
|
||||
* `COMMON_FJP` (default) use the common ForkJoinPool. Should use with [JRE 8u60 or higher](https://github.com/druid-io/druid/pull/4810#issuecomment-329922810). Older versions of the JRE may have worse performance than newer JRE versions.
|
||||
* `SINGLE_THREAD` Use a single-threaded executor.
|
||||
* `SAME_THREAD` Cache maintenance is done eagerly.
|
||||
|
||||
#### Metrics
|
||||
In addition to the normal cache metrics, the caffeine cache implementation also reports the following in both `total` and `delta`
|
||||
|
||||
|Metric|Description|Normal value|
|
||||
|------|-----------|------------|
|
||||
|`query/cache/caffeine/*/requests`|Count of hits or misses|hit + miss|
|
||||
|`query/cache/caffeine/*/loadTime`|Length of time caffeine spends loading new values (unused feature)|0|
|
||||
|`query/cache/caffeine/*/evictionBytes`|Size in bytes that have been evicted from the cache|Varies, should tune cache `sizeInBytes` so that `sizeInBytes`/`evictionBytes` is approximately the rate of cache churn you desire|
|
||||
|
||||
|
||||
#### Memcached
|
||||
|
||||
|
|
|
|||
|
|
@ -1,39 +0,0 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
|
||||
Druid Caffeine Cache
|
||||
--------------------
|
||||
|
||||
A highly performant local cache implementation for Druid based on [Caffeine](https://github.com/ben-manes/caffeine). Requires a JRE8u60 or higher
|
||||
|
||||
# Configuration
|
||||
Below are the configuration options known to this module:
|
||||
|
||||
|`runtime.properties`|Description|Default|
|
||||
|--------------------|-----------|-------|
|
||||
|`druid.cache.sizeInBytes`|The maximum size of the cache in bytes on heap.|None (unlimited)|
|
||||
|`druid.cache.expireAfter`|The time (in ms) after an access for which a cache entry may be expired|None (no time limit)|
|
||||
|`druid.cache.cacheExecutorFactory`|The executor factory to use for Caffeine maintenance. One of `COMMON_FJP`, `SINGLE_THREAD`, or `SAME_THREAD`|ForkJoinPool common pool (`COMMON_FJP`)|
|
||||
|`druid.cache.evictOnClose`|If a close of a namespace (ex: removing a segment from a node) should cause an eager eviction of associated cache values|`false`|
|
||||
|
||||
## `druid.cache.cacheExecutorFactory`
|
||||
|
||||
Here are the possible values for `druid.cache.cacheExecutorFactory`, which controls how maintenance tasks are run
|
||||
|
||||
* `COMMON_FJP` (default) use the common ForkJoinPool. Do NOT use this option unless you are running 8u60 or higher
|
||||
* `SINGLE_THREAD` Use a single-threaded executor
|
||||
* `SAME_THREAD` Cache maintenance is done eagerly
|
||||
|
||||
# Enabling
|
||||
|
||||
To enable the caffeine cache, include this module on the loadList and set `druid.cache.type` to `caffeine` in your properties.
|
||||
|
||||
# Metrics
|
||||
In addition to the normal cache metrics, the caffeine cache implementation also reports the following in both `total` and `delta`
|
||||
|
||||
|Metric|Description|Normal value|
|
||||
|------|-----------|------------|
|
||||
|`query/cache/caffeine/*/requests`|Count of hits or misses|hit + miss|
|
||||
|`query/cache/caffeine/*/loadTime`|Length of time caffeine spends loading new values (unused feature)|0|
|
||||
|`query/cache/caffeine/*/evictionBytes`|Size in bytes that have been evicted from the cache|Varies, should tune cache `sizeInBytes` so that `sizeInBytes`/`evictionBytes` is approximately the rate of cache churn you desire|
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
|
||||
<!--
|
||||
~ Licensed to Metamarkets Group Inc. (Metamarkets) under one
|
||||
~ or more contributor license agreements. See the NOTICE file
|
||||
~ distributed with this work for additional information
|
||||
~ regarding copyright ownership. Metamarkets licenses this file
|
||||
~ to you under the Apache License, Version 2.0 (the
|
||||
~ "License"); you may not use this file except in compliance
|
||||
~ with the License. You may obtain a copy of the License at
|
||||
~
|
||||
~ http://www.apache.org/licenses/LICENSE-2.0
|
||||
~
|
||||
~ Unless required by applicable law or agreed to in writing,
|
||||
~ software distributed under the License is distributed on an
|
||||
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
~ KIND, either express or implied. See the License for the
|
||||
~ specific language governing permissions and limitations
|
||||
~ under the License.
|
||||
-->
|
||||
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<groupId>io.druid.extensions</groupId>
|
||||
<artifactId>druid-caffeine-cache</artifactId>
|
||||
<name>druid-caffeine-cache</name>
|
||||
<description>Local cache implementation for Druid using Caffeine https://github.com/ben-manes/caffeine as the underlying implementation</description>
|
||||
|
||||
<parent>
|
||||
<groupId>io.druid</groupId>
|
||||
<artifactId>druid</artifactId>
|
||||
<version>0.11.0-SNAPSHOT</version>
|
||||
<relativePath>../../pom.xml</relativePath>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>io.druid</groupId>
|
||||
<artifactId>druid-api</artifactId>
|
||||
<scope>provided</scope>
|
||||
<version>${project.parent.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>io.druid</groupId>
|
||||
<artifactId>druid-server</artifactId>
|
||||
<version>${project.parent.version}</version>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.github.ben-manes.caffeine</groupId>
|
||||
<artifactId>caffeine</artifactId>
|
||||
<version>2.3.1</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>net.jpountz.lz4</groupId>
|
||||
<artifactId>lz4</artifactId>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
|
||||
<!-- Tests -->
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</project>
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
/*
|
||||
* Licensed to Metamarkets Group Inc. (Metamarkets) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. Metamarkets licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package io.druid.client.cache;
|
||||
|
||||
import com.fasterxml.jackson.databind.Module;
|
||||
import com.fasterxml.jackson.databind.module.SimpleModule;
|
||||
import com.google.common.collect.ImmutableList;
|
||||
import com.google.inject.Binder;
|
||||
import io.druid.initialization.DruidModule;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public class CaffeineDruidModule implements DruidModule
|
||||
{
|
||||
|
||||
@Override
|
||||
public void configure(Binder binder)
|
||||
{
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<? extends Module> getJacksonModules()
|
||||
{
|
||||
return ImmutableList.of(
|
||||
new SimpleModule("DruidCaffeineCache")
|
||||
.registerSubtypes(CaffeineCacheProvider.class)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -1 +0,0 @@
|
|||
io.druid.client.cache.CaffeineDruidModule
|
||||
7
pom.xml
7
pom.xml
|
|
@ -79,6 +79,7 @@
|
|||
<!-- Cannot update to AWS SDK 1.11+ because of Jackson incompatibility.
|
||||
Need to update Druid to use Jackson 2.6+ -->
|
||||
<aws.sdk.version>1.10.77</aws.sdk.version>
|
||||
<caffeine.version>2.5.5</caffeine.version>
|
||||
</properties>
|
||||
|
||||
<modules>
|
||||
|
|
@ -100,7 +101,6 @@
|
|||
<module>hll</module>
|
||||
<!-- Core extensions -->
|
||||
<module>extensions-core/avro-extensions</module>
|
||||
<module>extensions-core/caffeine-cache</module>
|
||||
<module>extensions-core/datasketches</module>
|
||||
<module>extensions-core/druid-kerberos</module>
|
||||
<module>extensions-core/hdfs-storage</module>
|
||||
|
|
@ -744,6 +744,11 @@
|
|||
<version>${calcite.version}</version>
|
||||
<type>test-jar</type>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.github.ben-manes.caffeine</groupId>
|
||||
<artifactId>caffeine</artifactId>
|
||||
<version>${caffeine.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.easymock</groupId>
|
||||
<artifactId>easymock</artifactId>
|
||||
|
|
|
|||
|
|
@ -185,6 +185,10 @@
|
|||
<groupId>it.unimi.dsi</groupId>
|
||||
<artifactId>fastutil</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.github.ben-manes.caffeine</groupId>
|
||||
<artifactId>caffeine</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- Tests -->
|
||||
<dependency>
|
||||
|
|
|
|||
|
|
@ -23,11 +23,12 @@ import com.fasterxml.jackson.annotation.JsonSubTypes;
|
|||
import com.fasterxml.jackson.annotation.JsonTypeInfo;
|
||||
import com.google.inject.Provider;
|
||||
|
||||
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type", defaultImpl = LocalCacheProvider.class)
|
||||
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type", defaultImpl = CaffeineCacheProvider.class)
|
||||
@JsonSubTypes(value = {
|
||||
@JsonSubTypes.Type(name = "local", value = LocalCacheProvider.class),
|
||||
@JsonSubTypes.Type(name = "memcached", value = MemcachedCacheProvider.class),
|
||||
@JsonSubTypes.Type(name = "hybrid", value = HybridCacheProvider.class)
|
||||
@JsonSubTypes.Type(name = "hybrid", value = HybridCacheProvider.class),
|
||||
@JsonSubTypes.Type(name = "caffeine", value = CaffeineCacheProvider.class)
|
||||
})
|
||||
public interface CacheProvider extends Provider<Cache>
|
||||
{
|
||||
|
|
|
|||
|
|
@ -19,9 +19,6 @@
|
|||
|
||||
package io.druid.client.cache;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonTypeName;
|
||||
|
||||
@JsonTypeName("caffeine")
|
||||
public class CaffeineCacheProvider extends CaffeineCacheConfig implements CacheProvider
|
||||
{
|
||||
@Override
|
||||
|
|
@ -24,7 +24,9 @@ import com.fasterxml.jackson.annotation.JsonProperty;
|
|||
import javax.validation.constraints.Min;
|
||||
|
||||
/**
|
||||
* Deprecated, use {@CaffeineCacheProvider}
|
||||
*/
|
||||
@Deprecated
|
||||
public class LocalCacheProvider implements CacheProvider
|
||||
{
|
||||
@JsonProperty
|
||||
|
|
|
|||
Loading…
Reference in New Issue