Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HBASE-20390: IMC Default Parameters for 2.0.0

This commit is contained in:
eshcar 2018-05-24 18:10:10 +03:00
parent bfab1e2f92
commit 079e08d7c5
4 changed files with 13 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java
View File

@ -43,12 +43,12 @@ import org.apache.hadoop.hbase.wal.WAL;
/** /**
* A memstore implementation which supports in-memory compaction. * A memstore implementation which supports in-memory compaction.
* A compaction pipeline is added between the active set and the snapshot data structures; * A compaction pipeline is added between the active set and the snapshot data structures;
* it consists of a list of kv-sets that are subject to compaction. * it consists of a list of segments that are subject to compaction.
* Like the snapshot, all pipeline components are read-only; updates only affect the active set. * Like the snapshot, all pipeline segments are read-only; updates only affect the active set.
* To ensure this property we take advantage of the existing blocking mechanism -- the active set * To ensure this property we take advantage of the existing blocking mechanism -- the active set
* is pushed to the pipeline while holding the region's updatesLock in exclusive mode. * is pushed to the pipeline while holding the region's updatesLock in exclusive mode.
* Periodically, a compaction is applied in the background to all pipeline components resulting * Periodically, a compaction is applied in the background to all pipeline segments resulting
* in a single read-only component. The ``old'' components are discarded when no scanner is reading * in a single read-only component. The ``old'' segments are discarded when no scanner is reading
* them. * them.
*/ */
@InterfaceAudience.Private @InterfaceAudience.Private
@ -62,7 +62,7 @@ public class CompactingMemStore extends AbstractMemStore {
// Default fraction of in-memory-flush size w.r.t. flush-to-disk size // Default fraction of in-memory-flush size w.r.t. flush-to-disk size
public static final String IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY = public static final String IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY =
"hbase.memstore.inmemoryflush.threshold.factor"; "hbase.memstore.inmemoryflush.threshold.factor";
private static final double IN_MEMORY_FLUSH_THRESHOLD_FACTOR_DEFAULT = 0.1; private static final double IN_MEMORY_FLUSH_THRESHOLD_FACTOR_DEFAULT = 0.014;
private static final Logger LOG = LoggerFactory.getLogger(CompactingMemStore.class); private static final Logger LOG = LoggerFactory.getLogger(CompactingMemStore.class);
private HStore store; private HStore store;

2
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactionStrategy.java
View File

@ -42,7 +42,7 @@ public abstract class MemStoreCompactionStrategy {
// The upper bound for the number of segments we store in the pipeline prior to merging. // The upper bound for the number of segments we store in the pipeline prior to merging.
public static final String COMPACTING_MEMSTORE_THRESHOLD_KEY = public static final String COMPACTING_MEMSTORE_THRESHOLD_KEY =
"hbase.hregion.compacting.pipeline.segments.limit"; "hbase.hregion.compacting.pipeline.segments.limit";
public static final int COMPACTING_MEMSTORE_THRESHOLD_DEFAULT = 4; public static final int COMPACTING_MEMSTORE_THRESHOLD_DEFAULT = 2;
/** /**
* Types of actions to be done on the pipeline upon MemStoreCompaction invocation. * Types of actions to be done on the pipeline upon MemStoreCompaction invocation.

4
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellFlatMapMemStore.java
View File

@ -848,8 +848,10 @@ public class TestCompactingToCellFlatMapMemStore extends TestCompactingMemStore
// set memstore to flat into CellChunkMap // set memstore to flat into CellChunkMap
MemoryCompactionPolicy compactionType = MemoryCompactionPolicy.BASIC; MemoryCompactionPolicy compactionType = MemoryCompactionPolicy.BASIC;
memstore.getConfiguration().setInt(MemStoreCompactionStrategy
.COMPACTING_MEMSTORE_THRESHOLD_KEY, 4);
memstore.getConfiguration().set(CompactingMemStore.COMPACTING_MEMSTORE_TYPE_KEY, memstore.getConfiguration().set(CompactingMemStore.COMPACTING_MEMSTORE_TYPE_KEY,
String.valueOf(compactionType)); String.valueOf(compactionType));
((MyCompactingMemStore) memstore).initiateType(compactionType, memstore.getConfiguration()); ((MyCompactingMemStore) memstore).initiateType(compactionType, memstore.getConfiguration());
((CompactingMemStore) memstore).setIndexType(CompactingMemStore.IndexType.CHUNK_MAP); ((CompactingMemStore) memstore).setIndexType(CompactingMemStore.IndexType.CHUNK_MAP);

7
src/main/asciidoc/_chapters/inmemory_compaction.adoc
View File

@ -52,7 +52,7 @@ In this section we describe how to enable Accordion and the available configurat
To enable in-memory compactions, set the _IN_MEMORY_COMPACTION_ attribute To enable in-memory compactions, set the _IN_MEMORY_COMPACTION_ attribute
on per column family where you want the behavior. The _IN_MEMORY_COMPACTION_ on per column family where you want the behavior. The _IN_MEMORY_COMPACTION_
attribute can have one of three values. attribute can have one of four values.
* _NONE_: No in-memory compaction. * _NONE_: No in-memory compaction.
* _BASIC_: Basic policy enables flushing and keeps a pipeline of flushes until we trip the pipeline maximum threshold and then we flush to disk. No in-memory compaction but can help throughput as data is moved from the profligate, native ConcurrentSkipListMap data-type to more compact (and efficient) data types. * _BASIC_: Basic policy enables flushing and keeps a pipeline of flushes until we trip the pipeline maximum threshold and then we flush to disk. No in-memory compaction but can help throughput as data is moved from the profligate, native ConcurrentSkipListMap data-type to more compact (and efficient) data types.
@ -94,11 +94,12 @@ to set the in-memory attribute to _NONE_).
When an in-memory flush happens is calculated by dividing the configured region flush size (Set in the table descriptor When an in-memory flush happens is calculated by dividing the configured region flush size (Set in the table descriptor
or read from _hbase.hregion.memstore.flush.size_) by the number of column families and then multiplying by or read from _hbase.hregion.memstore.flush.size_) by the number of column families and then multiplying by
_hbase.memstore.inmemoryflush.threshold.factor_ (default is 0.1). _hbase.memstore.inmemoryflush.threshold.factor_ (default is 0.014 - a little less than the size
of a chunk).
The number of flushes carried by the pipeline is monitored so as to fit within the bounds of memstore sizing The number of flushes carried by the pipeline is monitored so as to fit within the bounds of memstore sizing
but you can also set a maximum on the number of flushes total by setting but you can also set a maximum on the number of flushes total by setting
_hbase.hregion.compacting.pipeline.segments.limit_. Default is 4. _hbase.hregion.compacting.pipeline.segments.limit_. Default is 2.
When a column family Store is created, it says what memstore type is in effect. As of this writing When a column family Store is created, it says what memstore type is in effect. As of this writing
there is the old-school _DefaultMemStore_ which fills a _ConcurrentSkipListMap_ and then flushes there is the old-school _DefaultMemStore_ which fills a _ConcurrentSkipListMap_ and then flushes