Summary:
Porting one of our HBase cluster test tools (a single-process multi-threaded
load generator and verifier) from 0.89-fb to trunk.
I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that
it has some features that I have not tried yet (some kind of a kill test, and
some way to run HBase as multiple processes on one machine).
The main utility of this piece of code for us has been the HBaseClusterTest
command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a
load test in our five-node dev cluster testing, e.g.:
hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50💯20 -tn loadtest4
-read 100:10 -zk <zk_quorum_node> -bloom ROWCOL -compression LZO -key_window 5
-max_read_errors 10000 -num_keys 10000000000 -start_key 0
Test Plan:
Run this on a dev cluster. Run all unit tests.
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Reviewed By: nspiegelberg
CC: stack, nspiegelberg, mbautin, Karthik
Differential Revision: 549
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1211746 13f79535-47bb-0310-9956-ffa450edef68
Summary:
The metric blockcacheHitRatio is since the beginning of time. It would be nice
to calculate the block cache hit ratio for the past few minutes.
This patch remembers the blockcacheHitRatio for the last 5 periods by default.
Test Plan: unit test attached.
Reviewers: jgray, khemani, nspiegelberg, JIRA
Reviewed By: nspiegelberg
CC: nspiegelberg, dhruba
Differential Revision: 585
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1210640 13f79535-47bb-0310-9956-ffa450edef68
Summary:
HBASE-4463 Run more aggressive compactions during off peak hours
Increases the compact selection ratio from 1.3 to 5 at off-peak hours. This
will help utilize the available iops and bandwidth to decrease average num of
files per store. Only one such aggressive compaction is queued per store at any
point.
The number of iops on the disk and the top of the rack bandwidth utilization at
off peak hours is much lower than at peak hours depending on the application
usage pattern. We can utilize this knowledge to improve the performance of the
HBase cluster by increasing the compact selection ratio to a much larger value
during off-peak hours than otherwise - increasing hbase.hstore.compaction.ratio
(1.2 default) to hbase.hstore.compaction.ratio.offpeak (5 default). This will
help reduce the average number of files per store.
Test Plan: Started running the unit tests.
Reviewers: JIRA, Kannan, nspiegelberg, mbautin, stack
Reviewed By: nspiegelberg
CC: nspiegelberg, tedyu, lhofhansl, Karthik
Differential Revision: 471
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1208885 13f79535-47bb-0310-9956-ffa450edef68
Summary: With our cluster configuration setup it would be good to have a tool
that would merge HBase configuration files so that files appearing later in the
list would override properties specified in earlier files. This way we could
merge an application-specific configuration file with a cluster-specific
configuration file (with the latter overriding the former) and produce a single
HBase configuration file to install on the cluster.
Test Plan: Run the tool on two configuration files (common and
cluster-specific). Use the resulting configuration on a dev cluster.
Reviewers: todd, Karthik, tedyu, stack, JIRA
Reviewed By: Karthik
CC: Karthik, mbautin, todd
Differential Revision: 537
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1208832 13f79535-47bb-0310-9956-ffa450edef68
Summary: major compactions *try* to retain delete markers for the configured
time interval
Test Plan:
added a new test testDeleteMarkerLongevity() in TestStoreScanner
following tests passed
TestMemStore
TestStoreScanner
TestQueryMatcher
TestCompaction
Reviewers: jgray, dhruba, lhofhansl, Karthik, nspiegelberg
Reviewed By: nspiegelberg
CC: HBase Diffs Facebook Group, lhofhansl, khemani, nspiegelberg
Differential Revision: 321
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1208055 13f79535-47bb-0310-9956-ffa450edef68
Nicolas Spiegelberg
Michael Stack
Zhihong Yu
Doug Meil
Andrew Kyle Purtell
larsh
karthik
Todd Lipcon