be33a241ce
Disentangles flushes and compactions; flushes can proceed while a
compaction is happening. Also, don't compact unless we hit
compaction threshold: i.e. don't automatically compact on HRegion
startup so regions can come online the faster.
M src/contrib/hbase/conf/hbase-default.xml
(hbase.hregion.compactionThreashold): Moved to be a hstore property
as part of encapsulating compaction decision inside hstore.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseTestCase.java
Refactored. Moved here generalized content loading code that can
be shared by tests. Add to setup and teardown the setup and removal
of local test dir (if it exists).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestCompare.java
Added test of HStoreKey compare (It works other than one would at
first expect).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestSplit.java
Bulk of content loading code has been moved up into the parent class.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java
(tableExists): Restore to a check of if the asked-for table is in list of
tables. As it was, a check for tableExists would just wait on all timeouts
and retries to expire and then report table does not exist.. Fixed up
debug message listing regions of a table. Added protection against meta
table not having a COL_REGINFO (Seen in cluster testing -- probably a bug
in row removal).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
Loading store files, even if it was noticed that there was no corresponding
map file, was still counting file as valid. Also fix merger -- was
constructing MapFile.Reader directly rather than asking HStoreFile for
the reader (HStoreFile knows how to do MapFile references)
(rename): Added check that move succeeded and logging. In cluster-testing,
the hdfs move of compacted file into place has failed on occasion (Need
more info).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
Encapsulate ruling on whether a compaction should take place inside HStore.
Added reading of the compactionThreshold her. Compaction threshold is
currently just number of store files. Later may include other factors such
as count of reference files. Cleaned up debug messages around
reconstruction log. Removed compaction if size > 1 from constructor. Let
compaction happen after we've been deployed (Compactions that happen while
we are online can continue to take updates. Compaction in the constructor
puts off our being able to take in updates).
(close): Changed so it now returns set of store files. This used to be done
by calls to flush. Since flush and compaction have been disentangled, a
compaction can come in after flush and the list of files could be off.
Having it done by close, can be sure list of files is complete.
(flushCache): No longer returns set of store files. Added 'merging compaction'
where we pick an arbitrary store file from disk and merge into it the content
of memcache (Needs work).
(getAllMapFiles): Renamed getAllStoreFiles.
(needsCompaction): Added.
(compactHelper): Added passing of maximum sequence number if already
calculated. If compacting one file only, we used skip without rewriting
the info file. Fixed.
Refactored. Moved guts to new compact(outFile, listOfStores) method.
(compact, CompactionReader): Added overrides and interface to support
'merging compaction' that takes files and memcache. In compaction,
if we failed the move of the compacted file, all data had already been
deleted. Changing, so deletion happens after confirmed move of
compacted file.
(getFull): Fixed bug where NPE when read of maps came back null.
Revealed by our NOT compacting stores on startup. Meant could be two
backing stores one of which had no data regards queried key.
(getNMaps): Renamed countOfStoreFiles.
(toString): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreKey.java
Added comment on 'odd'-looking comparison.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Javadoc edit.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLogEdit.java
Only return first 128 bytes of value when toStringing (On cluster,
was returning complete web pages in log).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Removed confusing debug message (made sense once -- but not now).
Test rootRegionLocation for null before using it (can be null).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMemcache.java
Added comment that delete behavior needs study.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
Fixed merge so it doesn't do the incremental based off files
returned by flush. Instead all is done in the one go after
region closes (using files returned by close).
Moved duplicated code to new filesByFamily method.
(WriteState): Removed writesOngoing in favor of compacting and
flushing flags.
(flushCache): No longer returns list of files.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java
Fix javadoc.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@566459 13f79535-47bb-0310-9956-ffa450edef68
180 lines
6.6 KiB
XML
180 lines
6.6 KiB
XML
<?xml version="1.0"?>
|
|
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
|
<!--
|
|
/**
|
|
* Copyright 2007 The Apache Software Foundation
|
|
*
|
|
* Licensed to the Apache Software Foundation (ASF) under one
|
|
* or more contributor license agreements. See the NOTICE file
|
|
* distributed with this work for additional information
|
|
* regarding copyright ownership. The ASF licenses this file
|
|
* to you under the Apache License, Version 2.0 (the
|
|
* "License"); you may not use this file except in compliance
|
|
* with the License. You may obtain a copy of the License at
|
|
*
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
* See the License for the specific language governing permissions and
|
|
* limitations under the License.
|
|
*/
|
|
-->
|
|
<configuration>
|
|
<property>
|
|
<name>hbase.master</name>
|
|
<value>0.0.0.0:60000</value>
|
|
<description>The host and port that the HBase master runs at.
|
|
TODO: Support 'local' (All running in single context).
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.regionserver</name>
|
|
<value>0.0.0.0:60010</value>
|
|
<description>The host and port a HBase region server runs at.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.regionserver.class</name>
|
|
<value>org.apache.hadoop.hbase.HRegionInterface</value>
|
|
<description>An interface that is assignable to HRegionInterface. Used in HClient for
|
|
opening proxy to remote region server.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.rootdir</name>
|
|
<value>${hadoop.tmp.dir}/hbase</value>
|
|
<description>The directory shared by region servers.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.client.pause</name>
|
|
<value>30000</value>
|
|
<description>General client pause value. Used mostly as value to wait
|
|
before running a retry of a failed get, region lookup, etc.</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.client.retries.number</name>
|
|
<value>5</value>
|
|
<description>Maximum retries. Used as maximum for all retryable
|
|
operations such as fetching of the root region from root region
|
|
server, getting a cell's value, starting a row update, etc.
|
|
Default: 5.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.master.meta.thread.rescanfrequency</name>
|
|
<value>60000</value>
|
|
<description>How long the HMaster sleeps (in milliseconds) between scans of
|
|
the root and meta tables.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.master.lease.period</name>
|
|
<value>30000</value>
|
|
<description>HMaster server lease period in milliseconds. Default is
|
|
30 seconds. Region servers must report in within this period else
|
|
they are considered dead. On loaded cluster, may need to up this
|
|
period.</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.regionserver.lease.period</name>
|
|
<value>30000</value>
|
|
<description>HRegion server lease period in milliseconds. Default is
|
|
30 seconds. Clients must report in within this period else they are
|
|
considered dead.</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.server.thread.wakefrequency</name>
|
|
<value>10000</value>
|
|
<description>Time to sleep in between searches for work (in milliseconds).
|
|
Used as sleep interval by service threads such as META scanner and log roller.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.regionserver.handler.count</name>
|
|
<value>10</value>
|
|
<description>Count of RPC Server instances spun up on RegionServers
|
|
Same property is used by the HMaster for count of master handlers.
|
|
Default is 10.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.regionserver.msginterval</name>
|
|
<value>10000</value>
|
|
<description>Interval between messages from the RegionServer to HMaster
|
|
in milliseconds. Default is 15. Set this value low if you want unit
|
|
tests to be responsive.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.regionserver.maxlogentries</name>
|
|
<value>30000</value>
|
|
<description>Rotate the HRegion HLogs when count of entries exceeds this
|
|
value. Default: 30,000. Value is checked by a thread that runs every
|
|
hbase.server.thread.wakefrequency.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.hregion.memcache.flush.size</name>
|
|
<value>16777216</value>
|
|
<description>
|
|
A HRegion memcache will be flushed to disk if size of the memcache
|
|
exceeds this number of bytes. Value is checked by a thread that runs
|
|
every hbase.server.thread.wakefrequency.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.hregion.memcache.block.multiplier</name>
|
|
<value>2</value>
|
|
<description>
|
|
Block updates if memcache has hbase.hregion.block.memcache
|
|
time hbase.hregion.flush.size bytes. Useful preventing
|
|
runaway memcache during spikes in update traffic. Without an
|
|
upper-bound, memcache fills such that when it flushes the
|
|
resultant flush files take a long time to compact or split, or
|
|
worse, we OOME.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.hregion.max.filesize</name>
|
|
<value>67108864</value>
|
|
<description>
|
|
Maximum desired file size for an HRegion. If filesize exceeds
|
|
value + (value / 2), the HRegion is split in two. Default: 64M.
|
|
If too large, splits will take so long, clients timeout.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.hstore.compactionThreshold</name>
|
|
<value>3</value>
|
|
<description>
|
|
If more than this number of HStoreFiles in any one HStore
|
|
(one HStoreFile is written per flush of memcache) then a compaction
|
|
is run to rewrite all HStoreFiles files as one. Larger numbers
|
|
put off compaction but when it runs, it takes longer to complete.
|
|
During a compaction, updates cannot be flushed to disk. Long
|
|
compactions require memory sufficient to carry the logging of
|
|
all updates across the duration of the compaction.
|
|
|
|
If too large, clients timeout during compaction.
|
|
</description>
|
|
</property>
|
|
<property>
|
|
<name>hbase.regionserver.thread.splitcompactcheckfrequency</name>
|
|
<value>15000</value>
|
|
<description>How often a region server runs the split/compaction check.
|
|
</description>
|
|
</property>
|
|
|
|
<!-- HbaseShell Configurations -->
|
|
<property>
|
|
<name>hbaseshell.jline.bell.enabled</name>
|
|
<value>true</value>
|
|
<description>
|
|
if true, enable audible keyboard bells if an alert is required.
|
|
</description>
|
|
</property>
|
|
</configuration>
|