diff --git a/src/main/docbkx/book.xml b/src/main/docbkx/book.xml index b0273b2be1e..70e44811c3d 100644 --- a/src/main/docbkx/book.xml +++ b/src/main/docbkx/book.xml @@ -34,7 +34,7 @@ - + @@ -104,8 +104,7 @@ column contents:html is made up of the column family contents and html qualifier. - The colon character (:) delimits the column family from the + The colon character (:) delimits the column family from the column family qualifier. @@ -189,53 +188,58 @@ A namespace is a logical grouping of tables analogous to a database in relation database systems. This abstraction lays the groundwork for upcoming multi-tenancy related features: - Quota Management (HBASE-8410) - Restrict the amount of resources (ie - regions, tables) a namespace can consume. - Namespace Security Administration (HBASE-9206) - provide another - level of security administration for tenants. - Region server groups (HBASE-6721) - A namespace/table can be + Quota Management (HBASE-8410) - Restrict the amount of resources (ie + regions, tables) a namespace can consume. + Namespace Security Administration (HBASE-9206) - provide another + level of security administration for tenants. + Region server groups (HBASE-6721) - A namespace/table can be pinned onto a subset of regionservers thus guaranteeing a course level of - isolation. + isolation.
Namespace management A namespace can be created, removed or altered. Namespace membership is determined during - table creation by specifying a fully-qualified table name of the form: - - <table namespace>:<table qualifier> - - - Examples: - - + table creation by specifying a fully-qualified table name of the form: + + <table namespace>:<table qualifier> + + + + Examples + + #Create a namespace create_namespace 'my_ns' - + + #create my_table in my_ns namespace create 'my_ns:my_table', 'fam' - + + #drop namespace drop_namespace 'my_ns' - + + #alter namespace alter_namespace 'my_ns', {METHOD => 'set', 'PROPERTY_NAME' => 'PROPERTY_VALUE'} - - + +
Predefined namespaces There are two predefined special namespaces: - hbase - system namespace, used to contain hbase internal tables - default - tables with no explicit specified namespace will automatically - fall into this namespace. + hbase - system namespace, used to contain hbase internal tables + default - tables with no explicit specified namespace will automatically + fall into this namespace. - - Examples: + + Examples + #namespace=foo and table qualifier=bar create 'foo:bar', 'fam' @@ -243,7 +247,7 @@ create 'foo:bar', 'fam' #namespace=default and table qualifier=bar create 'bar', 'fam' - +
@@ -271,8 +275,8 @@ create 'bar', 'fam' courses:math are both members of the courses column family. The colon character (:) delimits the column family from the - column family qualifierColumn Family Qualifier. + >:) delimits the column family from the + column family qualifierColumn Family Qualifier. The column family prefix must be composed of printable characters. The qualifying tail, the column family qualifier, can be made of any @@ -651,9 +655,9 @@ htable.put(put);
ACID - 
See ACID Semantics.
+        See ACID Semantics.
             Lars Hofhansl has also written a note on
-            ACID in HBase.
+ ACID in HBase.
@@ -794,7 +798,7 @@ public static class MyMapper extends TableMapper<Text, Text> {
HBase MapReduce Read/Write Example The following is an example of using HBase both as a source and as a sink with MapReduce. - This example will simply copy data from one table to another. + This example will simply copy data from one table to another. Configuration config = HBaseConfiguration.create(); Job job = new Job(config,"ExampleReadWrite"); @@ -823,11 +827,11 @@ if (!b) { throw new IOException("error with job!"); } - An explanation is required of what TableMapReduceUtil is doing, especially with the reducer. + An explanation is required of what TableMapReduceUtil is doing, especially with the reducer. TableOutputFormat is being used as the outputFormat class, and several parameters are being set on the config (e.g., TableOutputFormat.OUTPUT_TABLE), as well as setting the reducer output key to ImmutableBytesWritable and reducer value to Writable. - These could be set by the programmer on the job and conf, but TableMapReduceUtil tries to make things easier. + These could be set by the programmer on the job and conf, but TableMapReduceUtil tries to make things easier. The following is the example mapper, which will create a Put and matching the input Result and emit it. Note: this is what the CopyTable utility does. @@ -854,7 +858,6 @@ public static class MyMapper extends TableMapper<ImmutableBytesWritable, Put& This is just an example, developers could choose not to use TableOutputFormat and connect to the target table themselves. -
HBase MapReduce Read/Write Example With Multi-Table Output @@ -962,8 +965,8 @@ if (!b) { throw new IOException("error with job!"); } - As stated above, the previous Mapper can run unchanged with this example. - As for the Reducer, it is a "generic" Reducer instead of extending TableMapper and emitting Puts. + As stated above, the previous Mapper can run unchanged with this example. + As for the Reducer, it is a "generic" Reducer instead of extending TableMapper and emitting Puts. public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable> { @@ -1082,18 +1085,18 @@ if (!b) { RDBMS can scale well, but only up to a point - specifically, the size of a single database server - and for the best performance requires specialized hardware and storage devices. HBase features of note are: - Strongly consistent reads/writes: HBase is not an "eventually consistent" DataStore. This - makes it very suitable for tasks such as high-speed counter aggregation. - Automatic sharding: HBase tables are distributed on the cluster via regions, and regions are - automatically split and re-distributed as your data grows. - Automatic RegionServer failover - Hadoop/HDFS Integration: HBase supports HDFS out of the box as its distributed file system. - MapReduce: HBase supports massively parallelized processing via MapReduce for using HBase as both - source and sink. - Java Client API: HBase supports an easy to use Java API for programmatic access. - Thrift/REST API: HBase also supports Thrift and REST for non-Java front-ends. - Block Cache and Bloom Filters: HBase supports a Block Cache and Bloom Filters for high volume query optimization. - Operational Management: HBase provides build-in web-pages for operational insight as well as JMX metrics. + Strongly consistent reads/writes: HBase is not an "eventually consistent" DataStore. This + makes it very suitable for tasks such as high-speed counter aggregation. + Automatic sharding: HBase tables are distributed on the cluster via regions, and regions are + automatically split and re-distributed as your data grows. + Automatic RegionServer failover + Hadoop/HDFS Integration: HBase supports HDFS out of the box as its distributed file system. + MapReduce: HBase supports massively parallelized processing via MapReduce for using HBase as both + source and sink. + Java Client API: HBase supports an easy to use Java API for programmatic access. + Thrift/REST API: HBase also supports Thrift and REST for non-Java front-ends. + Block Cache and Bloom Filters: HBase supports a Block Cache and Bloom Filters for high volume query optimization. + Operational Management: HBase provides build-in web-pages for operational insight as well as JMX metrics.
@@ -1140,15 +1143,15 @@ if (!b) { Key: - .META. region key (.META.,,1) + .META. region key (.META.,,1) Values: - info:regioninfo (serialized HRegionInfo - instance of .META.) - info:server (server:port of the RegionServer holding .META.) - info:serverstartcode (start-time of the RegionServer process holding .META.) + info:regioninfo (serialized HRegionInfo + instance of .META.) + info:server (server:port of the RegionServer holding .META.) + info:serverstartcode (start-time of the RegionServer process holding .META.) @@ -1158,16 +1161,16 @@ if (!b) { Key: - Region key of the format ([table],[region start key],[region id]) + Region key of the format ([table],[region start key],[region id]) Values: - info:regioninfo (serialized - HRegionInfo instance for this region) + info:regioninfo (serialized + HRegionInfo instance for this region) - info:server (server:port of the RegionServer containing this region) - info:serverstartcode (start-time of the RegionServer process containing this region) + info:server (server:port of the RegionServer containing this region) + info:serverstartcode (start-time of the RegionServer process containing this region) When a table is in the process of splitting two other columns will be created, info:splitA and info:splitB @@ -1351,7 +1354,7 @@ scan.setFilter(filter); See the Oracle JavaDoc for supported RegEx patterns in Java. -
SubstringComparator +
SubstringComparator SubstringComparator can be used to determine if a given substring exists in a value. The comparison is case-insensitive. @@ -1518,12 +1521,12 @@ rs.close();
Interface The methods exposed by HMasterInterface are primarily metadata-oriented methods: - Table (createTable, modifyTable, removeTable, enable, disable) - - ColumnFamily (addColumn, modifyColumn, removeColumn) - - Region (move, assign, unassign) - + Table (createTable, modifyTable, removeTable, enable, disable) + + ColumnFamily (addColumn, modifyColumn, removeColumn) + + Region (move, assign, unassign) + For example, when the HBaseAdmin method disableTable is invoked, it is serviced by the Master server. @@ -1551,9 +1554,9 @@ rs.close();
Interface The methods exposed by HRegionRegionInterface contain both data-oriented and region-maintenance methods: - Data (get, put, delete, next, etc.) + Data (get, put, delete, next, etc.) - Region (splitRegion, compactRegion, etc.) + Region (splitRegion, compactRegion, etc.) For example, when the HBaseAdmin method majorCompact is invoked on a table, the client is actually iterating through @@ -1597,14 +1600,14 @@ rs.close(); The Block Cache is an LRU cache that contains three levels of block priority to allow for scan-resistance and in-memory ColumnFamilies: - Single access priority: The first time a block is loaded from HDFS it normally has this priority and it will be part of the first group to be considered - during evictions. The advantage is that scanned blocks are more likely to get evicted than blocks that are getting more usage. + Single access priority: The first time a block is loaded from HDFS it normally has this priority and it will be part of the first group to be considered + during evictions. The advantage is that scanned blocks are more likely to get evicted than blocks that are getting more usage. - Multi access priority: If a block in the previous priority group is accessed - again, it upgrades to this priority. It is thus part of the second group considered - during evictions. - In-memory access priority: If the block's family was configured to be "in-memory", it will be part of this priority disregarding the number of times it - was accessed. Catalog tables are configured like this. This group is the last one considered during evictions. + Mutli access priority: If a block in the previous priority group is accessed again, it upgrades to this priority. It is thus part of the second group + considered during evictions. + + In-memory access priority: If the block's family was configured to be "in-memory", it will be part of this priority disregarding the number of times it + was accessed. Catalog tables are configured like this. This group is the last one considered during evictions. @@ -1628,27 +1631,27 @@ rs.close(); make the process blocking from the point where it loads new blocks. Here are some examples: - One region server with the default heap size (1GB) and the default block cache size will have 217MB of block cache available. + One region server with the default heap size (1GB) and the default block cache size will have 217MB of block cache available. - 20 region servers with the heap size set to 8GB and a default block cache size will have 34GB of block cache. + 20 region servers with the heap size set to 8GB and a default block cache size will have 34GB of block cache. - 100 region servers with the heap size set to 24GB and a block cache size of 0.5 will have about 1TB of block cache. + 100 region servers with the heap size set to 24GB and a block cache size of 0.5 will have about 1TB of block cache. Your data isn't the only resident of the block cache, here are others that you may have to take into account: - Catalog tables: The -ROOT- and .META. tables are forced into the block cache and have the in-memory priority which means that they are harder to evict. The former never uses - more than a few hundreds of bytes while the latter can occupy a few MBs (depending on the number of regions). + Catalog tables: The -ROOT- and .META. tables are forced into the block cache and have the in-memory priority which means that they are harder to evict. The former never uses + more than a few hundreds of bytes while the latter can occupy a few MBs (depending on the number of regions). - HFiles indexes: HFile is the file format that HBase uses to store data in HDFS and it contains a multi-layered index in order seek to the data without having to read the whole file. + HFiles indexes: HFile is the file format that HBase uses to store data in HDFS and it contains a multi-layered index in order seek to the data without having to read the whole file. The size of those indexes is a factor of the block size (64KB by default), the size of your keys and the amount of data you are storing. For big data sets it's not unusual to see numbers around - 1GB per region server, although not all of it will be in cache because the LRU will evict indexes that aren't used. + 1GB per region server, although not all of it will be in cache because the LRU will evict indexes that aren't used. - Keys: Taking into account only the values that are being stored is missing half the picture since every value is stored along with its keys - (row key, family, qualifier, and timestamp). See . + Keys: Taking into account only the values that are being stored is missing half the picture since every value is stored along with its keys + (row key, family, qualifier, and timestamp). See . - Bloom filters: Just like the HFile indexes, those data structures (when enabled) are stored in the LRU. + Bloom filters: Just like the HFile indexes, those data structures (when enabled) are stored in the LRU. Currently the recommended way to measure HFile indexes and bloom filters sizes is to look at the region server web UI and checkout the relevant metrics. For keys, @@ -1658,14 +1661,14 @@ rs.close(); but you need to process 1TB of data. One of the reasons is that the churn generated by the evictions will trigger more garbage collections unnecessarily. Here are two use cases: - Fully random reading pattern: This is a case where you almost never access the same row twice within a short amount of time such that the chance of hitting a cached block is close + Fully random reading pattern: This is a case where you almost never access the same row twice within a short amount of time such that the chance of hitting a cached block is close to 0. Setting block caching on such a table is a waste of memory and CPU cycles, more so that it will generate more garbage to pick up by the JVM. For more information on monitoring GC, - see . + see . - Mapping a table: In a typical MapReduce job that takes a table in input, every row will be read only once so there's no need to put them into the block cache. The Scan object has + Mapping a table: In a typical MapReduce job that takes a table in input, every row will be read only once so there's no need to put them into the block cache. The Scan object has the option of turning this off via the setCaching method (set it to false). You can still keep block caching turned on on this table if you need fast random read access. An example would be counting the number of rows in a table that serves live traffic, caching every block of that table would create massive churn and would surely evict data that's currently in use. - +
Offheap Block Cache @@ -1687,7 +1690,7 @@ rs.close(); This ensures that HBase has durable writes. Without WAL, there is the possibility of data loss in the case of a RegionServer failure before each MemStore is flushed and new StoreFiles are written. HLog is the HBase WAL implementation, and there is one HLog instance per RegionServer. - The WAL is in HDFS in /hbase/.logs/ with subdirectories per region. + The WAL is in HDFS in /hbase/.logs/ with subdirectories per region. For more general information about the concept of write ahead logs, see the Wikipedia Write-Ahead Log article. @@ -1759,13 +1762,14 @@ rs.close(); For a description of what HBase files look like when written to HDFS, see .
+ Considerations for Number of Regions In general, HBase is designed to run with a small (20-200) number of relatively large (5-20Gb) regions per server. The considerations for this are as follows:
Why cannot I have too many regions? Typically you want to keep your region count low on HBase for numerous reasons. Usually right around 100 regions per RegionServer has yielded the best results. - Here are some of the reasons below for keeping region count low: + Here are some of the reasons below for keeping region count low: MSLAB requires 2mb per memstore (that's 2mb per family per region). @@ -1795,12 +1799,13 @@ rs.close(); creating memory pressure or OOME on the RSs - + Another issue is the effect of the number of regions on mapreduce jobs; it is typical to have one mapper per HBase region. + Thus, hosting only 5 regions per RS may not be enough to get sufficient number of tasks for a mapreduce job, while 1000 regions will generate far too many tasks. + + See for configuration guidelines. +
- Another issue is the effect of the number of regions on mapreduce jobs; it is typical to have one mapper per HBase region. - Thus, hosting only 5 regions per RS may not be enough to get sufficient number of tasks for a mapreduce job, while 1000 regions will generate far too many tasks. - - See for configuration guidelines. +
@@ -1812,18 +1817,18 @@ rs.close(); Startup When HBase starts regions are assigned as follows (short version): - The Master invokes the AssignmentManager upon startup. + The Master invokes the AssignmentManager upon startup. - The AssignmentManager looks at the existing region assignments in META. + The AssignmentManager looks at the existing region assignments in META. - If the region assignment is still valid (i.e., if the RegionServer is still online) - then the assignment is kept. + If the region assignment is still valid (i.e., if the RegionServer is still online) + then the assignment is kept. - If the assignment is invalid, then the LoadBalancerFactory is invoked to assign the - region. The DefaultLoadBalancer will randomly assign the region to a RegionServer. + If the assignment is invalid, then the LoadBalancerFactory is invoked to assign the + region. The DefaultLoadBalancer will randomly assign the region to a RegionServer. - META is updated with the RegionServer assignment (if needed) and the RegionServer start codes - (start time of the RegionServer process) upon region opening by the RegionServer. + META is updated with the RegionServer assignment (if needed) and the RegionServer start codes + (start time of the RegionServer process) upon region opening by the RegionServer. @@ -1833,12 +1838,12 @@ rs.close(); Failover When a RegionServer fails (short version): - The regions immediately become unavailable because the RegionServer is down. + The regions immediately become unavailable because the RegionServer is down. - The Master will detect that the RegionServer has failed. + The Master will detect that the RegionServer has failed. - The region assignments will be considered invalid and will be re-assigned just - like the startup sequence. + The region assignments will be considered invalid and will be re-assigned just + like the startup sequence. @@ -1856,18 +1861,19 @@ rs.close();
Region-RegionServer Locality Over time, Region-RegionServer locality is achieved via HDFS block replication. - The HDFS client does the following by default when choosing locations to write replicas: + The HDFS client does the following by default when choosing locations to write replicas: - First replica is written to local node + First replica is written to local node - Second replica is written to a random node on another rack + Second replica is written to a random node on another rack - Third replica is written on the same rack as the second, but on a different node chosen randomly + Third replica is written on the same rack as the second, but on a different node chosen randomly - Subsequent replicas are written on random nodes on the cluster + Subsequent replicas are written on random nodes on the cluster See Replica Placement: The First Baby Steps on this page: HDFS Architecture - + + Thus, HBase eventually achieves locality for a region after a flush or a compaction. In a RegionServer failover situation a RegionServer may be assigned regions with non-local StoreFiles (because none of the replicas are local), however as new data is written @@ -1949,7 +1955,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( the SSTable file described in the BigTable [2006] paper and on Hadoop's tfile (The unit test suite and the compression harness were taken directly from tfile). - Schubert Zhang's blog post on HFile: A Block-Indexed File Format to Store Sorted Key-Value Pairs makes for a thorough introduction to HBase's hfile. Matteo Bertozzi has also put up a + Schubert Zhang's blog post on HFile: A Block-Indexed File Format to Store Sorted Key-Value Pairs makes for a thorough introduction to HBase's hfile. Matteo Bertozzi has also put up a helpful description, HBase I/O: HFile. For more information, see the HFile source code. @@ -1992,21 +1998,21 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( The KeyValue format inside a byte array is: - keylength - valuelength - key - value + keylength + valuelength + key + value The Key is further decomposed as: - rowlength - row (i.e., the rowkey) - columnfamilylength - columnfamily - columnqualifier - timestamp - keytype (e.g., Put, Delete, DeleteColumn, DeleteFamily) + rowlength + row (i.e., the rowkey) + columnfamilylength + columnfamily + columnqualifier + timestamp + keytype (e.g., Put, Delete, DeleteColumn, DeleteFamily) KeyValue instances are not split across blocks. @@ -2016,37 +2022,38 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName(
Example To emphasize the points above, examine what happens with two Puts for two different columns for the same row: - Put #1: rowkey=row1, cf:attr1=value1 - Put #2: rowkey=row1, cf:attr2=value2 + Put #1: rowkey=row1, cf:attr1=value1 + Put #2: rowkey=row1, cf:attr2=value2 Even though these are for the same row, a KeyValue is created for each column: Key portion for Put #1: - rowlength ------------> 4 - row -----------------> row1 - columnfamilylength ---> 2 - columnfamily --------> cf - columnqualifier ------> attr1 - timestamp -----------> server time of Put - keytype -------------> Put + rowlength ------------> 4 + row -----------------> row1 + columnfamilylength ---> 2 + columnfamily --------> cf + columnqualifier ------> attr1 + timestamp -----------> server time of Put + keytype -------------> Put Key portion for Put #2: - rowlength ------------> 4 - row -----------------> row1 - columnfamilylength ---> 2 - columnfamily --------> cf - columnqualifier ------> attr2 - timestamp -----------> server time of Put - keytype -------------> Put + rowlength ------------> 4 + row -----------------> row1 + columnfamilylength ---> 2 + columnfamily --------> cf + columnqualifier ------> attr2 + timestamp -----------> server time of Put + keytype -------------> Put + It is critical to understand that the rowkey, ColumnFamily, and column (aka columnqualifier) are embedded within + the KeyValue instance. The longer these identifiers are, the bigger the KeyValue is.
- It is critical to understand that the rowkey, ColumnFamily, and column (aka columnqualifier) are embedded within - the KeyValue instance. The longer these identifiers are, the bigger the KeyValue is. +
Compaction @@ -2078,16 +2085,16 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( Important knobs: - hbase.store.compaction.ratio Ratio used in compaction - file selection algorithm (default 1.2f). - hbase.hstore.compaction.min (.90 hbase.hstore.compactionThreshold) (files) Minimum number - of StoreFiles per Store to be selected for a compaction to occur (default 2). - hbase.hstore.compaction.max (files) Maximum number of StoreFiles to compact per minor compaction (default 10). - hbase.hstore.compaction.min.size (bytes) + hbase.store.compaction.ratio Ratio used in compaction + file selection algorithm (default 1.2f). + hbase.hstore.compaction.min (.90 hbase.hstore.compactionThreshold) (files) Minimum number + of StoreFiles per Store to be selected for a compaction to occur (default 2). + hbase.hstore.compaction.max (files) Maximum number of StoreFiles to compact per minor compaction (default 10). + hbase.hstore.compaction.min.size (bytes) Any StoreFile smaller than this setting with automatically be a candidate for compaction. Defaults to - hbase.hregion.memstore.flush.size (128 mb). - hbase.hstore.compaction.max.size (.92) (bytes) - Any StoreFile larger than this setting with automatically be excluded from compaction (default Long.MAX_VALUE). + hbase.hregion.memstore.flush.size (128 mb). + hbase.hstore.compaction.max.size (.92) (bytes) + Any StoreFile larger than this setting with automatically be excluded from compaction (default Long.MAX_VALUE). The minor compaction StoreFile selection logic is size based, and selects a file for compaction when the file @@ -2096,26 +2103,27 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName(
Minor Compaction File Selection - Example #1 (Basic Example) - This example mirrors an example from the unit test TestCompactSelection. + This example mirrors an example from the unit test TestCompactSelection. - hbase.store.compaction.ratio = 1.0f - hbase.hstore.compaction.min = 3 (files) > - hbase.hstore.compaction.max = 5 (files) > - hbase.hstore.compaction.min.size = 10 (bytes) > - hbase.hstore.compaction.max.size = 1000 (bytes) > + hbase.store.compaction.ratio = 1.0f + hbase.hstore.compaction.min = 3 (files) + hbase.hstore.compaction.max = 5 (files) + hbase.hstore.compaction.min.size = 10 (bytes) + hbase.hstore.compaction.max.size = 1000 (bytes) + The following StoreFiles exist: 100, 50, 23, 12, and 12 bytes apiece (oldest to newest). With the above parameters, the files that would be selected for minor compaction are 23, 12, and 12. Why? - 100 --> No, because sum(50, 23, 12, 12) * 1.0 = 97. - 50 --> No, because sum(23, 12, 12) * 1.0 = 47. - 23 --> Yes, because sum(12, 12) * 1.0 = 24. - 12 --> Yes, because the previous file has been included, and because this - does not exceed the the max-file limit of 5 - 12 --> Yes, because the previous file had been included, and because this - does not exceed the the max-file limit of 5. + 100 --> No, because sum(50, 23, 12, 12) * 1.0 = 97. + 50 --> No, because sum(23, 12, 12) * 1.0 = 47. + 23 --> Yes, because sum(12, 12) * 1.0 = 24. + 12 --> Yes, because the previous file has been included, and because this + does not exceed the the max-file limit of 5 + 12 --> Yes, because the previous file had been included, and because this + does not exceed the the max-file limit of 5.
@@ -2123,11 +2131,11 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( Minor Compaction File Selection - Example #2 (Not Enough Files To Compact) This example mirrors an example from the unit test TestCompactSelection. - hbase.store.compaction.ratio = 1.0f - hbase.hstore.compaction.min = 3 (files) > - hbase.hstore.compaction.max = 5 (files) > - hbase.hstore.compaction.min.size = 10 (bytes) > - hbase.hstore.compaction.max.size = 1000 (bytes) > + hbase.store.compaction.ratio = 1.0f + hbase.hstore.compaction.min = 3 (files) + hbase.hstore.compaction.max = 5 (files) + hbase.hstore.compaction.min.size = 10 (bytes) + hbase.hstore.compaction.max.size = 1000 (bytes) The following StoreFiles exist: 100, 25, 12, and 12 bytes apiece (oldest to newest). @@ -2135,35 +2143,35 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( Why? - 100 --> No, because sum(25, 12, 12) * 1.0 = 47 - 25 --> No, because sum(12, 12) * 1.0 = 24 - 12 --> No. Candidate because sum(12) * 1.0 = 12, there are only 2 files to compact and that is less than the threshold of 3 - 12 --> No. Candidate because the previous StoreFile was, but there are not enough files to compact + 100 --> No, because sum(25, 12, 12) * 1.0 = 47 + 25 --> No, because sum(12, 12) * 1.0 = 24 + 12 --> No. Candidate because sum(12) * 1.0 = 12, there are only 2 files to compact and that is less than the threshold of 3 + 12 --> No. Candidate because the previous StoreFile was, but there are not enough files to compact
-
+
Minor Compaction File Selection - Example #3 (Limiting Files To Compact) This example mirrors an example from the unit test TestCompactSelection. - hbase.store.compaction.ratio = 1.0f - hbase.hstore.compaction.min = 3 (files) > - hbase.hstore.compaction.max = 5 (files) > - hbase.hstore.compaction.min.size = 10 (bytes) > - hbase.hstore.compaction.max.size = 1000 (bytes) > + hbase.store.compaction.ratio = 1.0f + hbase.hstore.compaction.min = 3 (files) + hbase.hstore.compaction.max = 5 (files) + hbase.hstore.compaction.min.size = 10 (bytes) + hbase.hstore.compaction.max.size = 1000 (bytes) The following StoreFiles exist: 7, 6, 5, 4, 3, 2, and 1 bytes apiece (oldest to newest). With the above parameters, the files that would be selected for minor compaction are 7, 6, 5, 4, 3. Why? - 7 --> Yes, because sum(6, 5, 4, 3, 2, 1) * 1.0 = 21. Also, 7 is less than the min-size - 6 --> Yes, because sum(5, 4, 3, 2, 1) * 1.0 = 15. Also, 6 is less than the min-size. - 5 --> Yes, because sum(4, 3, 2, 1) * 1.0 = 10. Also, 5 is less than the min-size. - 4 --> Yes, because sum(3, 2, 1) * 1.0 = 6. Also, 4 is less than the min-size. - 3 --> Yes, because sum(2, 1) * 1.0 = 3. Also, 3 is less than the min-size. - 2 --> No. Candidate because previous file was selected and 2 is less than the min-size, but the max-number of files to compact has been reached. - 1 --> No. Candidate because previous file was selected and 1 is less than the min-size, but max-number of files to compact has been reached. + 7 --> Yes, because sum(6, 5, 4, 3, 2, 1) * 1.0 = 21. Also, 7 is less than the min-size + 6 --> Yes, because sum(5, 4, 3, 2, 1) * 1.0 = 15. Also, 6 is less than the min-size. + 5 --> Yes, because sum(4, 3, 2, 1) * 1.0 = 10. Also, 5 is less than the min-size. + 4 --> Yes, because sum(3, 2, 1) * 1.0 = 6. Also, 4 is less than the min-size. + 3 --> Yes, because sum(2, 1) * 1.0 = 3. Also, 3 is less than the min-size. + 2 --> No. Candidate because previous file was selected and 2 is less than the min-size, but the max-number of files to compact has been reached. + 1 --> No. Candidate because previous file was selected and 1 is less than the min-size, but max-number of files to compact has been reached.
@@ -2187,11 +2195,11 @@ This feature is fully compatible with default compactions - it can be enabled fo
When to use You might want to consider using this feature if you have: - -large regions (in that case, you can get the positive effect of much smaller regions without additional memstore and region management overhead); or - +large regions (in that case, you can get the positive effect of much smaller regions without additional memstore and region management overhead); or + + non-uniform row keys, e.g. time dimension in a key (in that case, only the stripes receiving the new keys will keep compacting - old data will not compact as much, or at all). - + According to perf testing performed, in these case the read performance can improve somewhat, and the read and write performance variability due to compactions is greatly reduced. There's overall perf improvement on large, non-uniform row key regions (hash-prefixed timestamp key) over long term. All of these performance gains are best realized when table is already large. In future, the perf improvement might also extend to region splits. @@ -2226,21 +2234,23 @@ Based on your region sizing, you might want to also change your stripe sizing. B You can improve this pattern for your data. You should generally aim at stripe size of at least 1Gb, and about 8-12 stripes for uniform row keys - so, for example if your regions are 30 Gb, 12x2.5Gb stripes might be a good idea. The settings are as follows: - + SettingNotes hbase.store.stripe.initialStripeCount -Initial stripe count to create. You can use it as follows: - - +Initial stripe count to create. You can use it as follows: + + for relatively uniform row keys, if you know the approximate target number of stripes from the above, you can avoid some splitting overhead by starting w/several stripes (2, 5, 10...). Note that if the early data is not representative of overall row key distribution, this will not be as efficient. - + + for existing tables with lots of data, you can use this to pre-split stripes. - -for e.g. hash-prefixed sequential keys, with more than one hash prefix per region, you know that some pre-splitting makes sense. - + + +for e.g. hash-prefixed sequential keys, with more than one hash prefix per region, you know that some pre-splitting makes sense. + hbase.store.stripe.sizeToSplit @@ -2252,7 +2262,7 @@ Maximum stripe size before it's split. You can use this in conjunction with the The number of new stripes to create when splitting one. The default is 2, and is good for most cases. For non-uniform row keys, you might experiment with increasing the number somewhat (3-4), to isolate the arriving updates into narrower slice of the region with just one split instead of several. -
+
Memstore sizing @@ -2354,7 +2364,7 @@ All the settings that apply to normal compactions (file size limits, etc.) apply where importtsv or your MapReduce job put its results, and the table name to import into. For example: - $ hadoop jar hbase-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /user/todd/myoutput mytable + $ hadoop jar hbase-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /user/todd/myoutput mytable The -c config-file option can be used to specify a file containing the appropriate hbase parameters (e.g., hbase-site.xml) if @@ -2374,7 +2384,7 @@ All the settings that apply to normal compactions (file size limits, etc.) apply For more information about the referenced utilities, see and . - See How-to: Use HBase Bulk Loading, and Why + See How-to: Use HBase Bulk Loading, and Why for a recent blog on current state of bulk loading.
@@ -2422,12 +2432,11 @@ All the settings that apply to normal compactions (file size limits, etc.) apply - FAQ - + General When should I use HBase? @@ -2488,7 +2497,7 @@ All the settings that apply to normal compactions (file size limits, etc.) apply - + Where can I learn about the rest of the configuration options? @@ -2629,7 +2638,7 @@ identifying mode and a multi-phase read-write repair mode.
Running hbck to identify inconsistencies -To check to see if your HBase cluster has corruptions, run hbck against your HBase cluster: +To check to see if your HBase cluster has corruptions, run hbck against your HBase cluster: $ ./bin/hbase hbck @@ -2645,9 +2654,9 @@ listing of all the splits present in all the tables. $ ./bin/hbase hbck -details -If you just want to know if some tables are corrupted, you can limit hbck to identify inconsistencies +If you just want to know if some tables are corrupted, you can limit hbck to identify inconsistencies in only specific tables. For example the following command would only attempt to check table -TableFoo and TableBar. The benefit is that hbck will run in less time. +TableFoo and TableBar. The benefit is that hbck will run in less time. $ ./bin/hbase hbck TableFoo TableBar @@ -2662,12 +2671,12 @@ the hbck tool enabled with automatic repair options. There are two invariants that when violated create inconsistencies in HBase: - HBase’s region consistency invariant is satisfied if every region is assigned and + HBase’s region consistency invariant is satisfied if every region is assigned and deployed on exactly one region server, and all places where this state kept is in -accordance. +accordance. - HBase’s table integrity invariant is satisfied if for each table, every possible row key -resolves to exactly one region. + HBase’s table integrity invariant is satisfied if for each table, every possible row key +resolves to exactly one region. @@ -2698,11 +2707,11 @@ Region consistency requires that the HBase instance has the state of the region (.regioninfo files), the region’s row in the .META. table., and region’s deployment/assignments on region servers and the master in accordance. Options for repairing region consistency include: - -fixAssignments (equivalent to the 0.90 -fix option) repairs unassigned, incorrectly -assigned or multiply assigned regions. + -fixAssignments (equivalent to the 0.90 -fix option) repairs unassigned, incorrectly +assigned or multiply assigned regions. - -fixMeta which removes meta rows when corresponding regions are not present in -HDFS and adds new meta rows if they regions are present in HDFS while not in META. + -fixMeta which removes meta rows when corresponding regions are not present in + HDFS and adds new meta rows if they regions are present in HDFS while not in META. To fix deployment and assignment problems you can run this command: @@ -2710,48 +2719,48 @@ HDFS and adds new meta rows if they regions are present in HDFS while not in MET $ ./bin/hbase hbck -fixAssignments -To fix deployment and assignment problems as well as repairing incorrect meta rows you can -run this command:. +To fix deployment and assignment problems as well as repairing incorrect meta rows you can +run this command: $ ./bin/hbase hbck -fixAssignments -fixMeta -There are a few classes of table integrity problems that are low risk repairs. The first two are +There are a few classes of table integrity problems that are low risk repairs. The first two are degenerate (startkey == endkey) regions and backwards regions (startkey > endkey). These are automatically handled by sidelining the data to a temporary directory (/hbck/xxxx). -The third low-risk class is hdfs region holes. This can be repaired by using the: +The third low-risk class is hdfs region holes. This can be repaired by using the: - -fixHdfsHoles option for fabricating new empty regions on the file system. -If holes are detected you can use -fixHdfsHoles and should include -fixMeta and -fixAssignments to make the new region consistent. + -fixHdfsHoles option for fabricating new empty regions on the file system. +If holes are detected you can use -fixHdfsHoles and should include -fixMeta and -fixAssignments to make the new region consistent. $ ./bin/hbase hbck -fixAssignments -fixMeta -fixHdfsHoles -Since this is a common operation, we’ve added a the -repairHoles flag that is equivalent to the -previous command: +Since this is a common operation, we’ve added a the -repairHoles flag that is equivalent to the +previous command: $ ./bin/hbase hbck -repairHoles -If inconsistencies still remain after these steps, you most likely have table integrity problems -related to orphaned or overlapping regions. +If inconsistencies still remain after these steps, you most likely have table integrity problems +related to orphaned or overlapping regions.
Region Overlap Repairs -Table integrity problems can require repairs that deal with overlaps. This is a riskier operation +Table integrity problems can require repairs that deal with overlaps. This is a riskier operation because it requires modifications to the file system, requires some decision making, and may require some manual steps. For these repairs it is best to analyze the output of a hbck -details run so that you isolate repairs attempts only upon problems the checks identify. Because this is riskier, there are safeguard that should be used to limit the scope of the repairs. WARNING: This is a relatively new and have only been tested on online but idle HBase instances (no reads/writes). Use at your own risk in an active production environment! -The options for repairing table integrity violations include: +The options for repairing table integrity violations include: - -fixHdfsOrphans option for “adopting” a region directory that is missing a region -metadata file (the .regioninfo file). + -fixHdfsOrphans option for “adopting” a region directory that is missing a region +metadata file (the .regioninfo file). - -fixHdfsOverlaps ability for fixing overlapping regions + -fixHdfsOverlaps ability for fixing overlapping regions -When repairing overlapping regions, a region’s data can be modified on the file system in two +When repairing overlapping regions, a region’s data can be modified on the file system in two ways: 1) by merging regions into a larger region or 2) by sidelining regions by moving data to “sideline” directory where data could be restored later. Merging a large number of regions is technically correct but could result in an extremely large region that requires series of costly @@ -2760,58 +2769,58 @@ that overlap with the most other regions (likely the largest ranges) so that mer a more reasonable scale. Since these sidelined regions are already laid out in HBase’s native directory and HFile format, they can be restored by using HBase’s bulk load mechanism. The default safeguard thresholds are conservative. These options let you override the default -thresholds and to enable the large region sidelining feature. +thresholds and to enable the large region sidelining feature. - -maxMerge <n> maximum number of overlapping regions to merge + -maxMerge <n> maximum number of overlapping regions to merge - -sidelineBigOverlaps if more than maxMerge regions are overlapping, sideline attempt -to sideline the regions overlapping with the most other regions. + -sidelineBigOverlaps if more than maxMerge regions are overlapping, sideline attempt +to sideline the regions overlapping with the most other regions. - -maxOverlapsToSideline <n> if sidelining large overlapping regions, sideline at most n -regions. + -maxOverlapsToSideline <n> if sidelining large overlapping regions, sideline at most n +regions. -Since often times you would just want to get the tables repaired, you can use this option to turn -on all repair options: +Since often times you would just want to get the tables repaired, you can use this option to turn +on all repair options: - -repair includes all the region consistency options and only the hole repairing table -integrity options. + -repair includes all the region consistency options and only the hole repairing table +integrity options. -Finally, there are safeguards to limit repairs to only specific tables. For example the following -command would only attempt to check and repair table TableFoo and TableBar. - +Finally, there are safeguards to limit repairs to only specific tables. For example the following +command would only attempt to check and repair table TableFoo and TableBar. + $ ./bin/hbase hbck -repair TableFoo TableBar - +
Special cases: Meta is not properly assigned -There are a few special cases that hbck can handle as well. +There are a few special cases that hbck can handle as well. Sometimes the meta table’s only region is inconsistently assigned or deployed. In this case -there is a special -fixMetaOnly option that can try to fix meta assignments. - +there is a special -fixMetaOnly option that can try to fix meta assignments. + $ ./bin/hbase hbck -fixMetaOnly -fixAssignments - +
Special cases: HBase version file is missing -HBase’s data on the file system requires a version file in order to start. If this flie is missing, you +HBase’s data on the file system requires a version file in order to start. If this flie is missing, you can use the -fixVersionFile option to fabricating a new HBase version file. This assumes that -the version of hbck you are running is the appropriate version for the HBase cluster. +the version of hbck you are running is the appropriate version for the HBase cluster.
Special case: Root and META are corrupt. -The most drastic corruption scenario is the case where the ROOT or META is corrupted and +The most drastic corruption scenario is the case where the ROOT or META is corrupted and HBase will not start. In this case you can use the OfflineMetaRepair tool create new ROOT and META regions and tables. This tool assumes that HBase is offline. It then marches through the existing HBase home directory, loads as much information from region metadata files (.regioninfo files) as possible from the file system. If the region metadata has proper table integrity, it sidelines the original root and meta table directories, and builds new ones with pointers to the region directories and their -data. - +data. + $ ./bin/hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair - -NOTE: This tool is not as clever as uberhbck but can be used to bootstrap repairs that uberhbck + +NOTE: This tool is not as clever as uberhbck but can be used to bootstrap repairs that uberhbck can complete. -If the tool succeeds you should be able to start hbase and run online repairs if necessary. +If the tool succeeds you should be able to start hbase and run online repairs if necessary.
Special cases: Offline split parent @@ -3022,15 +3031,12 @@ hbase> describe 't1'
HFile format version 1 overview As we will be discussing the changes we are making to the HFile format, it is useful to give a short overview of the previous (HFile version 1) format. An HFile in the existing format is structured as follows: - + HFile Version 1 - - HFile Version 1 - - + Image courtesy of Lars George, hbase-architecture-101-storage.html. @@ -3065,15 +3071,11 @@ hbase> describe 't1' The version of HBase introducing the above features reads both version 1 and 2 HFiles, but only writes version 2 HFiles. A version 2 HFile is structured as follows: - + HFile Version 2 - - HFile Version 2 - - @@ -3341,7 +3343,7 @@ Comparator class used for Bloom filter keys, a UTF>8 encoded string stored usi
File Info format in versions 1 and 2 - The file info block is a serialized HbaseMapWritable (essentially a map from byte arrays to byte arrays) with the following keys, among others. StoreFile-level logic adds more keys to this. + The file info block is a serialized HbaseMapWritable (essentially a map from byte arrays to byte arrays) with the following keys, among others. StoreFile-level logic adds more keys to this. @@ -3479,8 +3481,8 @@ Comparator class used for Bloom filter keys, a UTF>8 encoded string stored usi In HBASE-7845, we generate a new key that is lexicographically larger than the last key of the previous block and lexicographically equal or smaller than the start key of the current block. While actual keys can potentially be very long, this "fake key" or "virtual key" can be much shorter. For example, if the stop key of previous block is "the quick brown fox", the start key of current block is "the who", we could use "the r" as our virtual key in our hfile index. There are two benefits to this: -
having shorter keys reduces the hfile index size, (allowing us to keep more indexes in memory), and
 -
using something closer to the end key of the previous block allows us to avoid a potential extra IO when the target key lives in between the "virtual key" and the key of the first element in the target block.
 + having shorter keys reduces the hfile index size, (allowing us to keep more indexes in memory), and + using something closer to the end key of the previous block allows us to avoid a potential extra IO when the target key lives in between the "virtual key" and the key of the first element in the target block. This optimization (implemented by the getShortMidpointKey method) is inspired by LevelDB's ByteWiseComparatorImpl::FindShortestSeparator() and FindShortSuccessor().
@@ -3490,10 +3492,10 @@ Comparator class used for Bloom filter keys, a UTF>8 encoded string stored usi
HBase Videos Introduction to HBase - Introduction to HBase by Todd Lipcon (Chicago Data Summit 2011). - - Introduction to HBase by Todd Lipcon (2010). - + Introduction to HBase by Todd Lipcon (Chicago Data Summit 2011). + + Introduction to HBase by Todd Lipcon (2010). + Building Real Time Services at Facebook with HBase by Jonathan Gray (Hadoop World 2011). @@ -3520,8 +3522,8 @@ Comparator class used for Bloom filter keys, a UTF>8 encoded string stored usi
HBase Sites Cloudera's HBase Blog has a lot of links to useful HBase information. - CAP Confusion is a relevant entry for background information on - distributed storage systems. + CAP Confusion is a relevant entry for background information on + distributed storage systems. @@ -3543,14 +3545,14 @@ Comparator class used for Bloom filter keys, a UTF>8 encoded string stored usi HBase History - 2006: BigTable paper published by Google. - - 2006 (end of year): HBase development starts. - - 2008: HBase becomes Hadoop sub-project. - - 2010: HBase becomes Apache top-level project. - + 2006: BigTable paper published by Google. + + 2006 (end of year): HBase development starts. + + 2008: HBase becomes Hadoop sub-project. + + 2010: HBase becomes Apache top-level project. + diff --git a/src/main/docbkx/case_studies.xml b/src/main/docbkx/case_studies.xml index 067a8b59204..15169a8aa43 100644 --- a/src/main/docbkx/case_studies.xml +++ b/src/main/docbkx/case_studies.xml @@ -1,13 +1,13 @@ - Apache HBase Case Studies -
- Overview - This chapter will describe a variety of performance and troubleshooting case studies that can +
+ Overview + This chapter will describe a variety of performance and troubleshooting case studies that can provide a useful blueprint on diagnosing Apache HBase cluster issues. - For more information on Performance and Troubleshooting, see and . - -
- -
- Schema Design - See the schema design case studies here: - - -
- -
- Performance/Troubleshooting - + For more information on Performance and Troubleshooting, see and . + +
+ +
+ Schema Design + See the schema design case studies here: + + +
+ +
+ Performance/Troubleshooting +
Case Study #1 (Performance Issue On A Single Node)
Scenario Following a scheduled reboot, one data node began exhibiting unusual behavior. Routine MapReduce - jobs run against HBase tables which regularly completed in five or six minutes began taking 30 or 40 minutes - to finish. These jobs were consistently found to be waiting on map and reduce tasks assigned to the troubled data node - (e.g., the slow map tasks all had the same Input Split). - The situation came to a head during a distributed copy, when the copy was severely prolonged by the lagging node. - -
+ jobs run against HBase tables which regularly completed in five or six minutes began taking 30 or 40 minutes + to finish. These jobs were consistently found to be waiting on map and reduce tasks assigned to the troubled data node + (e.g., the slow map tasks all had the same Input Split). + The situation came to a head during a distributed copy, when the copy was severely prolonged by the lagging node. + +
Hardware Datanodes: - - Two 12-core processors - Six Enerprise SATA disks - 24GB of RAM - Two bonded gigabit NICs - + + Two 12-core processors + Six Enerprise SATA disks + 24GB of RAM + Two bonded gigabit NICs + Network: - - 10 Gigabit top-of-rack switches - 20 Gigabit bonded interconnects between racks. - + + 10 Gigabit top-of-rack switches + 20 Gigabit bonded interconnects between racks. +
Hypotheses -
HBase "Hot Spot" Region - We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table, - where uneven key-space distribution can funnel a huge number of requests to a single HBase region, bombarding the RegionServer - process and cause slow response time. Examination of the HBase Master status page showed that the number of HBase requests to the - troubled node was almost zero. Further, examination of the HBase logs showed that there were no region splits, compactions, or other region transitions - in progress. This effectively ruled out a "hot spot" as the root cause of the observed slowness. +
HBase "Hot Spot" Region + We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table, + where uneven key-space distribution can funnel a huge number of requests to a single HBase region, bombarding the RegionServer + process and cause slow response time. Examination of the HBase Master status page showed that the number of HBase requests to the + troubled node was almost zero. Further, examination of the HBase logs showed that there were no region splits, compactions, or other region transitions + in progress. This effectively ruled out a "hot spot" as the root cause of the observed slowness.
-
HBase Region With Non-Local Data - Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus - forcing HDFS to request data blocks from other servers over the network. Examination of the datanode logs showed that there were very - few blocks being requested over the network, indicating that the HBase region was correctly assigned, and that the majority of the necessary - data was located on the node. This ruled out the possibility of non-local data causing a slowdown. +
HBase Region With Non-Local Data + Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus + forcing HDFS to request data blocks from other servers over the network. Examination of the datanode logs showed that there were very + few blocks being requested over the network, indicating that the HBase region was correctly assigned, and that the majority of the necessary + data was located on the node. This ruled out the possibility of non-local data causing a slowdown.
-
Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk +
Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk After concluding that the Hadoop and HBase were not likely to be the culprits, we moved on to troubleshooting the datanode's hardware. - Java, by design, will periodically scan its entire memory space to do garbage collection. If system memory is heavily overcommitted, the Linux - kernel may enter a vicious cycle, using up all of its resources swapping Java heap back and forth from disk to RAM as Java tries to run garbage - collection. Further, a failing hard disk will often retry reads and/or writes many times before giving up and returning an error. This can manifest - as high iowait, as running processes wait for reads and writes to complete. Finally, a disk nearing the upper edge of its performance envelope will - begin to cause iowait as it informs the kernel that it cannot accept any more data, and the kernel queues incoming data into the dirty write pool in memory. - However, using vmstat(1) and free(1), we could see that no swap was being used, and the amount of disk IO was only a few kilobytes per second. + Java, by design, will periodically scan its entire memory space to do garbage collection. If system memory is heavily overcommitted, the Linux + kernel may enter a vicious cycle, using up all of its resources swapping Java heap back and forth from disk to RAM as Java tries to run garbage + collection. Further, a failing hard disk will often retry reads and/or writes many times before giving up and returning an error. This can manifest + as high iowait, as running processes wait for reads and writes to complete. Finally, a disk nearing the upper edge of its performance envelope will + begin to cause iowait as it informs the kernel that it cannot accept any more data, and the kernel queues incoming data into the dirty write pool in memory. + However, using vmstat(1) and free(1), we could see that no swap was being used, and the amount of disk IO was only a few kilobytes per second.
-
Slowness Due To High Processor Usage +
Slowness Due To High Processor Usage Next, we checked to see whether the system was performing slowly simply due to very high computational load. top(1) showed that the system load - was higher than normal, but vmstat(1) and mpstat(1) showed that the amount of processor being used for actual computation was low. + was higher than normal, but vmstat(1) and mpstat(1) showed that the amount of processor being used for actual computation was low.
-
Network Saturation (The Winner) +
Network Saturation (The Winner) Since neither the disks nor the processors were being utilized heavily, we moved on to the performance of the network interfaces. The datanode had two - gigabit ethernet adapters, bonded to form an active-standby interface. ifconfig(8) showed some unusual anomalies, namely interface errors, overruns, framing errors. - While not unheard of, these kinds of errors are exceedingly rare on modern hardware which is operating as it should: - + gigabit ethernet adapters, bonded to form an active-standby interface. ifconfig(8) showed some unusual anomalies, namely interface errors, overruns, framing errors. + While not unheard of, these kinds of errors are exceedingly rare on modern hardware which is operating as it should: + $ /sbin/ifconfig bond0 bond0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 inet addr:10.x.x.x Bcast:10.x.x.255 Mask:255.255.255.0 @@ -118,9 +118,9 @@ RX bytes:2416328868676 (2.4 TB) TX bytes:3464991094001 (3.4 TB) These errors immediately lead us to suspect that one or more of the ethernet interfaces might have negotiated the wrong line speed. This was confirmed both by running an ICMP ping - from an external host and observing round-trip-time in excess of 700ms, and by running ethtool(8) on the members of the bond interface and discovering that the active interface - was operating at 100Mbs/, full duplex. - + from an external host and observing round-trip-time in excess of 700ms, and by running ethtool(8) on the members of the bond interface and discovering that the active interface + was operating at 100Mbs/, full duplex. + $ sudo ethtool eth0 Settings for eth0: Supported ports: [ TP ] @@ -148,44 +148,44 @@ Wake-on: g Current message level: 0x00000003 (3) Link detected: yes - - In normal operation, the ICMP ping round trip time should be around 20ms, and the interface speed and duplex should read, "1000MB/s", and, "Full", respectively. - -
-
-
Resolution - After determining that the active ethernet adapter was at the incorrect speed, we used the ifenslave(8) command to make the standby interface - the active interface, which yielded an immediate improvement in MapReduce performance, and a 10 times improvement in network throughput: - - On the next trip to the datacenter, we determined that the line speed issue was ultimately caused by a bad network cable, which was replaced. - -
-
+ + In normal operation, the ICMP ping round trip time should be around 20ms, and the interface speed and duplex should read, "1000MB/s", and, "Full", respectively. + +
+
+
Resolution + After determining that the active ethernet adapter was at the incorrect speed, we used the ifenslave(8) command to make the standby interface + the active interface, which yielded an immediate improvement in MapReduce performance, and a 10 times improvement in network throughput: + + On the next trip to the datacenter, we determined that the line speed issue was ultimately caused by a bad network cable, which was replaced. + +
+
Case Study #2 (Performance Research 2012) Investigation results of a self-described "we're not sure what's wrong, but it seems slow" problem. - http://gbif.blogspot.com/2012/03/hbase-performance-evaluation-continued.html + http://gbif.blogspot.com/2012/03/hbase-performance-evaluation-continued.html
- +
Case Study #3 (Performance Research 2010)) - Investigation results of general cluster performance from 2010. Although this research is on an older version of the codebase, this writeup - is still very useful in terms of approach. - http://hstack.org/hbase-performance-testing/ + Investigation results of general cluster performance from 2010. Although this research is on an older version of the codebase, this writeup + is still very useful in terms of approach. + http://hstack.org/hbase-performance-testing/
- +
Case Study #4 (xcievers Config) Case study of configuring xceivers, and diagnosing errors from mis-configurations. - http://www.larsgeorge.com/2012/03/hadoop-hbase-and-xceivers.html + http://www.larsgeorge.com/2012/03/hadoop-hbase-and-xceivers.html See also .
- -
- - + +
+ + diff --git a/src/main/docbkx/configuration.xml b/src/main/docbkx/configuration.xml index 45bc157742b..25e1f1376ad 100644 --- a/src/main/docbkx/configuration.xml +++ b/src/main/docbkx/configuration.xml @@ -173,7 +173,7 @@ needed for servers to pick up changes (caveat dynamic config. to be described la A useful read setting config on you hadoop cluster is Aaron Kimballs' Configuration + xlink:href="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/">Configuration Parameters: What can you just ignore? @@ -527,8 +527,7 @@ homed on the node h-24-30.example.com. Now skip to for how to start and verify your pseudo-distributed install. - See Pseudo-distributed - mode extras for notes on how to start extra Masters and + See for notes on how to start extra Masters and RegionServers when running pseudo-distributed. @@ -695,11 +694,7 @@ homed on the node h-24-30.example.com. bin/start-hbase.sh - Run the above from the - - HBASE_HOME - - directory. + Run the above from the HBASE_HOME directory. You should now have a running HBase instance. HBase logs can be found in the logs subdirectory. Check them out @@ -771,7 +766,79 @@ stopping hbase............... Shutdown can take a moment to The generated file is a docbook section with a glossary in it--> - + + +
+ + + This file is fallback content. If you are seeing this, something is wrong with the build of the HBase documentation or you are doing pre-build verification. + + + The file hbase-default.xml is generated as part of + the build of the hbase site. See the hbase pom.xml. + The generated file is a docbook glossary. + +
+ IDs that are auto-generated and cause validation errors if not present + + Each of these is a reference to a configuration file parameter which will cause an error if you are using the fallback content here. This is a dirty dirty hack. + +
+ fail.fast.expired.active.master + +
+
+ "hbase.hregion.memstore.flush.size" + +
+
+ hbase.hstore.bytes.per.checksum + +
+
+ hbase.online.schema.update.enable + +
+
+ hbase.regionserver.global.memstore.size + +
+
+ hbase.hregion.max.filesize + +
+
+ hbase.hstore.BlockingStoreFiles + +
+
+ hfile.block.cache.size + +
+
+ copy.table + +
+
+ hbase.hstore.checksum.algorithm + +
+
+ hbase.zookeeper.useMulti + +
+
+ hbase.hregion.memstore.block.multiplier + +
+
+ hbase.regionserver.global.memstore.size.lower.limit + +
+
+
+ +
diff --git a/src/main/docbkx/developer.xml b/src/main/docbkx/developer.xml index dd469b65d60..cedc6a5d399 100644 --- a/src/main/docbkx/developer.xml +++ b/src/main/docbkx/developer.xml @@ -118,8 +118,8 @@ git clone git://github.com/apache/hbase.git Maven Classpath Variable The M2_REPO classpath variable needs to be set up for the project. This needs to be set to your local Maven repository, which is usually ~/.m2/repository - If this classpath variable is not configured, you will see compile errors in Eclipse like this... - +If this classpath variable is not configured, you will see compile errors in Eclipse like this: + Description Resource Path Location Type The project cannot be built until build path errors are resolved hbase Unknown Java Problem Unbound classpath variable: 'M2_REPO/asm/asm/3.1/asm-3.1.jar' in project 'hbase' hbase Build path Build Path Problem @@ -223,51 +223,52 @@ mvn compile -Dcompile-protobuf -Dprotoc.path=/opt/local/bin/protoc poms when you build. For now, just be aware of the difference between HBase 1.x builds and those of HBase 0.96-0.98. Below we will come back to this difference when we list out build instructions. -
+ + Publishing to maven requires you sign the artifacts you want to upload. To have the build do this for you, you need to make sure you have a properly configured settings.xml in your local repository under .m2. Here is my ~/.m2/settings.xml. - <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" + - <servers> - <!- To publish a snapshot of some part of Maven --> - <server> - <id>apache.snapshots.https</id> - <username>YOUR_APACHE_ID - </username> - <password>YOUR_APACHE_PASSWORD - </password> - </server> - <!-- To publish a website using Maven --> - <!-- To stage a release of some part of Maven --> - <server> - <id>apache.releases.https</id> - <username>YOUR_APACHE_ID - </username> - <password>YOUR_APACHE_PASSWORD - </password> - </server> - </servers> - <profiles> - <profile> - <id>apache-release</id> - <properties> - <gpg.keyname>YOUR_KEYNAME</gpg.keyname> - <!--Keyname is something like this ... 00A5F21E... do gpg --list-keys to find it--> - <gpg.passphrase>YOUR_KEY_PASSWORD - </gpg.passphrase> - </properties> - </profile> - </profiles> -</settings> + + + + apache.snapshots.https + YOUR_APACHE_ID + + YOUR_APACHE_PASSWORD + + + + + + apache.releases.https + YOUR_APACHE_ID + + YOUR_APACHE_PASSWORD + + + + + + apache-release + + YOUR_KEYNAME + + YOUR_KEY_PASSWORD + + + + +]]> You must use maven 3.0.x (Check by running mvn -version). - +
Making a Release Candidate I'll explain by running through the process. See later in this section for more detail on particular steps. @@ -501,17 +502,17 @@ HBase have a character not usually seen in other projects. dependency tree).
Running Tests in other Modules - If the module you are developing in has no other dependencies on other HBase modules, then - you can cd into that module and just run: + If the module you are developing in has no other dependencies on other HBase modules, then + you can cd into that module and just run: mvn test - which will just run the tests IN THAT MODULE. If there are other dependencies on other modules, + which will just run the tests IN THAT MODULE. If there are other dependencies on other modules, then you will have run the command from the ROOT HBASE DIRECTORY. This will run the tests in the other modules, unless you specify to skip the tests in that module. For instance, to skip the tests in the hbase-server module, - you would run: + you would run: mvn clean test -PskipServerTests - from the top level directory to run all the tests in modules other than hbase-server. Note that you + from the top level directory to run all the tests in modules other than hbase-server. Note that you can specify to skip tests in multiple modules as well as just for a single module. For example, to skip - the tests in hbase-server and hbase-common, you would run: + the tests in hbase-server and hbase-common, you would run: mvn clean test -PskipServerTests -PskipCommonTests Also, keep in mind that if you are running tests in the hbase-server module you will need to apply the maven profiles discussed in to get the tests to run properly. @@ -541,7 +542,7 @@ The first three categories, small, medium, and large are for tests run when you type $ mvn test; i.e. these three categorizations are for HBase unit tests. The integration category is for not for unit tests but for integration tests. These are run when you invoke $ mvn verify. Integration tests -are described in integration tests section and will not be discussed further +are described in and will not be discussed further in this section on HBase unit tests. Apache HBase uses a patched maven surefire plugin and maven profiles to implement @@ -579,7 +580,7 @@ the developer machine as well.
Integration Tests<indexterm><primary>IntegrationTests</primary></indexterm> Integration tests are system level tests. See -integration tests section for more info. + for more info.
@@ -704,17 +705,17 @@ should not impact these resources, it's worth checking these log lines General rules -As much as possible, tests should be written as category small tests. +As much as possible, tests should be written as category small tests. -All tests must be written to support parallel execution on the same machine, hence they should not use shared resources as fixed ports or fixed file names. +All tests must be written to support parallel execution on the same machine, hence they should not use shared resources as fixed ports or fixed file names. -Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. +Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. -Tests can be written with HBaseTestingUtility. -This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster. +Tests can be written with HBaseTestingUtility. +This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster.
@@ -722,19 +723,19 @@ This class offers helper functions to create a temp directory and do the cleanup Categories and execution time -All tests must be categorized, if not they could be skipped. +All tests must be categorized, if not they could be skipped. -All tests should be written to be as fast as possible. +All tests should be written to be as fast as possible. -Small category tests should last less than 15 seconds, and must not have any side effect. +Small category tests should last less than 15 seconds, and must not have any side effect. -Medium category tests should last less than 50 seconds. +Medium category tests should last less than 50 seconds. -Large category tests should last less than 3 minutes. This should ensure a good parallelization for people using it, and ease the analysis when the test fails. +Large category tests should last less than 3 minutes. This should ensure a good parallelization for people using it, and ease the analysis when the test fails.
@@ -862,17 +863,17 @@ are running other tests. -ChaosMonkey defines Action's and Policy's. Actions are sequences of events. We have at least the following actions: +ChaosMonkey defines Action's and Policy's. Actions are sequences of events. We have at least the following actions: -Restart active master (sleep 5 sec) -Restart random regionserver (sleep 5 sec) -Restart random regionserver (sleep 60 sec) -Restart META regionserver (sleep 5 sec) -Restart ROOT regionserver (sleep 5 sec) -Batch restart of 50% of regionservers (sleep 5 sec) -Rolling restart of 100% of regionservers (sleep 5 sec) +Restart active master (sleep 5 sec) +Restart random regionserver (sleep 5 sec) +Restart random regionserver (sleep 60 sec) +Restart META regionserver (sleep 5 sec) +Restart ROOT regionserver (sleep 5 sec) +Batch restart of 50% of regionservers (sleep 5 sec) +Rolling restart of 100% of regionservers (sleep 5 sec) - + Policies on the other hand are responsible for executing the actions based on a strategy. The default policy is to execute a random action every minute based on predefined action weights. ChaosMonkey executes predefined named policies until it is stopped. More than one @@ -881,11 +882,12 @@ policy can be active at any time. To run ChaosMonkey as a standalone tool deploy your HBase cluster as usual. ChaosMonkey uses the configuration -from the bin/hbase script, thus no extra configuration needs to be done. You can invoke the ChaosMonkey by running: +from the bin/hbase script, thus no extra configuration needs to be done. You can invoke the ChaosMonkey by running: bin/hbase org.apache.hadoop.hbase.util.ChaosMonkey - + This will output smt like: - + + 12/11/19 23:21:57 INFO util.ChaosMonkey: Using ChaosMonkey Policy: class org.apache.hadoop.hbase.util.ChaosMonkey$PeriodicRandomActionPolicy, period:60000 12/11/19 23:21:57 INFO util.ChaosMonkey: Sleeping for 26953 to add jitter 12/11/19 23:22:24 INFO util.ChaosMonkey: Performing action: Restart active master @@ -921,8 +923,8 @@ This will output smt like: 12/11/19 23:24:26 INFO hbase.ClusterManager: Executed remote command, exit code:0 , output:starting regionserver, logging to /homes/enis/code/hbase-0.94/bin/../logs/hbase-enis-regionserver-rs3.example.com.out 12/11/19 23:24:27 INFO util.ChaosMonkey: Started region server:rs3.example.com,60020,1353367027826. Reported num of rs:6 - - + + As you can see from the log, ChaosMonkey started the default PeriodicRandomActionPolicy, which is configured with all the available actions, and ran RestartActiveMaster and RestartRandomRs actions. ChaosMonkey tool, if run from command line, will keep on running until the process is killed.
@@ -958,7 +960,7 @@ mvn compile The above will build against whatever explicit hadoop 1.x version we have in our pom.xml as our '1.0' version. Tests may not all pass so you may need to pass -DskipTests unless you are inclined to fix the failing tests. - + 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:${compat.module}:test-jar with value '${compat.module}' does not match a valid id pattern You will see ERRORs like the above title if you pass the default profile; e.g. if you pass hadoop.profile=1.1 when building 0.96 or @@ -1001,12 +1003,12 @@ pecularity that is probably fixable but we've not spent the time trying to figur
Jira Priorities The following is a guideline on setting Jira issue priorities: - Blocker: Should only be used if the issue WILL cause data loss or cluster instability reliably. - Critical: The issue described can cause data loss or cluster instability in some cases. - Major: Important but not tragic issues, like updates to the client API that will add a lot of much-needed functionality or significant - bugs that need to be fixed but that don't cause data loss. - Minor: Useful enhancements and annoying but not damaging bugs. - Trivial: Useful enhancements but generally cosmetic. + Blocker: Should only be used if the issue WILL cause data loss or cluster instability reliably. + Critical: The issue described can cause data loss or cluster instability in some cases. + Major: Important but not tragic issues, like updates to the client API that will add a lot of much-needed functionality or significant + bugs that need to be fixed but that don't cause data loss. + Minor: Useful enhancements and annoying but not damaging bugs. + Trivial: Useful enhancements but generally cosmetic.
@@ -1161,10 +1163,9 @@ pecularity that is probably fixable but we've not spent the time trying to figur Please submit one patch-file per Jira. For example, if multiple files are changed make sure the selected resource when generating the patch is a directory. Patch files can reflect changes in multiple files. - Generating patches using git: - -$ git diff --no-prefix > HBASE_XXXX.patch - + Generating patches using git: +$ git diff --no-prefix > HBASE_XXXX.patch + Don't forget the 'no-prefix' option; and generate the diff from the root directory of project Make sure you review for code style. @@ -1283,11 +1284,10 @@ Bar bar = foo.getBar(); <--- imagine there's an extra space(s) after the
Javadoc - This is also a very common feedback item. Don't forget Javadoc! + This is also a very common feedback item. Don't forget Javadoc! Javadoc warnings are checked during precommit. If the precommit tool gives you a '-1', please fix the javadoc issue. Your patch won't be committed if it adds such warnings. -
Findbugs @@ -1345,25 +1345,25 @@ Bar bar = foo.getBar(); <--- imagine there's an extra space(s) after the - Do not delete the old patch file + Do not delete the old patch file - version your new patch file using a simple scheme like this: - HBASE-{jira number}-{version}.patch - e.g: - HBASE_XXXX-v2.patch + version your new patch file using a simple scheme like this: + HBASE-{jira number}-{version}.patch + e.g: + HBASE_XXXX-v2.patch - 'Cancel Patch' on JIRA.. bug status will change back to Open + 'Cancel Patch' on JIRA.. bug status will change back to Open - Attach new patch file (e.g. HBASE_XXXX-v2.patch) using 'Files --> Attach' + Attach new patch file (e.g. HBASE_XXXX-v2.patch) using 'Files --> Attach' - Click on 'Submit Patch'. Now the bug status will say 'Patch Available'. + Click on 'Submit Patch'. Now the bug status will say 'Patch Available'. - Committers will review the patch. Rinse and repeat as many times as needed :-) + Committers will review the patch. Rinse and repeat as many times as needed :-)
@@ -1372,32 +1372,32 @@ Bar bar = foo.getBar(); <--- imagine there's an extra space(s) after the At times you may want to break a big change into mulitple patches. Here is a sample work-flow using git - patch 1: + patch 1: - $ git diff --no-prefix > HBASE_XXXX-1.patch + $ git diff --no-prefix > HBASE_XXXX-1.patch - patch 2: + patch 2: - create a new git branch - $ git checkout -b my_branch + create a new git branch + $ git checkout -b my_branch - save your work - $ git add file1 file2 - $ git commit -am 'saved after HBASE_XXXX-1.patch' - now you have your own branch, that is different from remote master branch + save your work + $ git add file1 file2 + $ git commit -am 'saved after HBASE_XXXX-1.patch' + now you have your own branch, that is different from remote master branch - make more changes... + make more changes... - create second patch - $ git diff --no-prefix > HBASE_XXXX-2.patch + create second patch + $ git diff --no-prefix > HBASE_XXXX-2.patch diff --git a/src/main/docbkx/external_apis.xml b/src/main/docbkx/external_apis.xml index 9e3ed372798..8ad1c8bcfd2 100644 --- a/src/main/docbkx/external_apis.xml +++ b/src/main/docbkx/external_apis.xml @@ -27,8 +27,8 @@ */ --> Apache HBase External APIs - This chapter will cover access to Apache HBase either through non-Java languages, or through custom protocols. - + This chapter will cover access to Apache HBase either through non-Java languages, or through custom protocols. +
Non-Java Languages Talking to the JVM Currently the documentation on this topic in the @@ -172,7 +172,7 @@ Example4: =, 'substring:abc123' will match everything that begins with the substring "abc123"
-
Example PHP Client Program that uses the Filter Language +
Example PHP Client Program that uses the Filter Language <? $_SERVER['PHP_ROOT'] = realpath(dirname(__FILE__).'/..'); require_once $_SERVER['PHP_ROOT'].'/flib/__flib.php'; @@ -205,7 +205,7 @@ - + “(RowFilter (=, ‘binary:Row 1’) AND TimeStampsFilter (74689, 89734)) OR @@ -216,7 +216,7 @@ 1) The key-value pair must be in a column that is lexicographically >= abc and < xyz  - + @@ -228,7 +228,7 @@
-
Individual Filter Syntax +
Individual Filter Syntax KeyOnlyFilter diff --git a/src/main/docbkx/getting_started.xml b/src/main/docbkx/getting_started.xml index cd47284c6e3..c99adf8fbce 100644 --- a/src/main/docbkx/getting_started.xml +++ b/src/main/docbkx/getting_started.xml @@ -1,13 +1,13 @@ - Getting Started - +
Introduction - + will get you up and - running on a single-node, standalone instance of HBase. + running on a single-node, standalone instance of HBase.
- +
Quick Start - + This guide describes setup of a standalone HBase instance. It will - run against the local filesystem. In later sections we will take you through - how to run HBase on Apache Hadoop's HDFS, a distributed filesystem. This section - shows you how to create a table in HBase, inserting - rows into your new HBase table via the HBase shell, and then cleaning - up and shutting down your standalone, local filesystem-based HBase instance. The below exercise - should take no more than ten minutes (not including download time). + run against the local filesystem. In later sections we will take you through + how to run HBase on Apache Hadoop's HDFS, a distributed filesystem. This section + shows you how to create a table in HBase, inserting + rows into your new HBase table via the HBase shell, and then cleaning + up and shutting down your standalone, local filesystem-based HBase instance. The below exercise + should take no more than ten minutes (not including download time). Local Filesystem and Durability - Using HBase with a LocalFileSystem does not currently guarantee durability. + Using HBase with a LocalFileSystem does not currently guarantee durability. The HDFS local filesystem implementation will lose edits if files are not properly closed -- which is very likely to happen when experimenting with a new download. - You need to run HBase on HDFS to ensure all writes are preserved. Running - against the local filesystem though will get you off the ground quickly and get you - familiar with how the general system works so lets run with it for now. See - and its associated issues for more details. + You need to run HBase on HDFS to ensure all writes are preserved. Running + against the local filesystem though will get you off the ground quickly and get you + familiar with how the general system works so lets run with it for now. See + and its associated issues for more details. - Loopback IP - - The below advice is for hbase-0.94.x and older versions only. We believe this fixed in hbase-0.96.0 and beyond -(let us know if we have it wrong). There should be no need of the below modification to /etc/hosts in -later versions of HBase. - - HBase expects the loopback IP address to be 127.0.0.1. Ubuntu and some other distributions, - for example, will default to 127.0.1.1 and this will cause problems for you - See Why does HBase care about /etc/hosts? for detail.. - - /etc/hosts should look something like this: - + Loopback IP + The below advice is for hbase-0.94.x and older versions only. We believe this fixed in hbase-0.96.0 and beyond + (let us know if we have it wrong). There should be no need of the below modification to /etc/hosts in + later versions of HBase. + + HBase expects the loopback IP address to be 127.0.0.1. Ubuntu and some other distributions, + for example, will default to 127.0.1.1 and this will cause problems for you + See Why does HBase care about /etc/hosts? for detail.. + + /etc/hosts should look something like this: + 127.0.0.1 localhost 127.0.0.1 ubuntu.ubuntu-domain ubuntu - - - - + + + +
Download and unpack the latest stable release. - + Choose a download site from this list of Apache Download - Mirrors. Click on the suggested top link. This will take you to a - mirror of HBase Releases. Click on the folder named - stable and then download the file that ends in - .tar.gz to your local filesystem; e.g. - hbase-0.94.2.tar.gz. - + xlink:href="http://www.apache.org/dyn/closer.cgi/hbase/">Apache Download + Mirrors. Click on the suggested top link. This will take you to a + mirror of HBase Releases. Click on the folder named + stable and then download the file that ends in + .tar.gz to your local filesystem; e.g. + hbase-0.94.2.tar.gz. + Decompress and untar your download and then change into the - unpacked directory. - + unpacked directory. + $ tar xfz hbase-.tar.gz $ cd hbase- - + At this point, you are ready to start HBase. But before starting - it, edit conf/hbase-site.xml, the file you write - your site-specific configurations into. Set - hbase.rootdir, the directory HBase writes data to, - and hbase.zookeeper.property.dataDir, the directory - ZooKeeper writes its data too: -<?xml version="1.0"?> + it, edit conf/hbase-site.xml, the file you write + your site-specific configurations into. Set + hbase.rootdir, the directory HBase writes data to, + and hbase.zookeeper.property.dataDir, the directory + ZooKeeper writes its data too: + <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> @@ -111,63 +110,63 @@ $ cd hbase- <value>/DIRECTORY/zookeeper</value> </property> </configuration> Replace DIRECTORY in the above with the - path to the directory you would have HBase and ZooKeeper write their data. By default, - hbase.rootdir is set to /tmp/hbase-${user.name} - and similarly so for the default ZooKeeper data location which means you'll lose all - your data whenever your server reboots unless you change it (Most operating systems clear - /tmp on restart). + path to the directory you would have HBase and ZooKeeper write their data. By default, + hbase.rootdir is set to /tmp/hbase-${user.name} + and similarly so for the default ZooKeeper data location which means you'll lose all + your data whenever your server reboots unless you change it (Most operating systems clear + /tmp on restart).
- +
Start HBase - + Now start HBase:$ ./bin/start-hbase.sh starting Master, logging to logs/hbase-user-master-example.org.out - + You should now have a running standalone HBase instance. In - standalone mode, HBase runs all daemons in the the one JVM; i.e. both - the HBase and ZooKeeper daemons. HBase logs can be found in the - logs subdirectory. Check them out especially if - it seems HBase had trouble starting. - + standalone mode, HBase runs all daemons in the the one JVM; i.e. both + the HBase and ZooKeeper daemons. HBase logs can be found in the + logs subdirectory. Check them out especially if + it seems HBase had trouble starting. + Is <application>java</application> installed? - + All of the above presumes a 1.6 version of Oracle - java is installed on your machine and - available on your path (See ); i.e. when you type - java, you see output that describes the - options the java program takes (HBase requires java 6). If this is not - the case, HBase will not start. Install java, edit - conf/hbase-env.sh, uncommenting the - JAVA_HOME line pointing it to your java install, then, - retry the steps above. + java is installed on your machine and + available on your path (See ); i.e. when you type + java, you see output that describes the + options the java program takes (HBase requires java 6). If this is not + the case, HBase will not start. Install java, edit + conf/hbase-env.sh, uncommenting the + JAVA_HOME line pointing it to your java install, then, + retry the steps above.
- +
Shell Exercises - + Connect to your running HBase via the shell. - + $ ./bin/hbase shell HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version: 0.90.0, r1001068, Fri Sep 24 13:55:42 PDT 2010 hbase(main):001:0> - + Type help and then - <RETURN> to see a listing of shell commands and - options. Browse at least the paragraphs at the end of the help emission - for the gist of how variables and command arguments are entered into the - HBase shell; in particular note how table names, rows, and columns, - etc., must be quoted. - - Create a table named test with a single column family named cf. - Verify its creation by listing all tables and then insert some - values. - + <RETURN> to see a listing of shell commands and + options. Browse at least the paragraphs at the end of the help emission + for the gist of how variables and command arguments are entered into the + HBase shell; in particular note how table names, rows, and columns, + etc., must be quoted. + + Create a table named test with a single column family named cf. + Verify its creation by listing all tables and then insert some + values. + hbase(main):003:0> create 'test', 'cf' 0 row(s) in 1.2200 seconds hbase(main):003:0> list 'test' @@ -179,59 +178,59 @@ hbase(main):005:0> put 'test', 'row2', 'cf:b', 'value2' 0 row(s) in 0.0370 seconds hbase(main):006:0> put 'test', 'row3', 'cf:c', 'value3' 0 row(s) in 0.0450 seconds - + Above we inserted 3 values, one at a time. The first insert is at - row1, column cf:a with a value of - value1. Columns in HBase are comprised of a column family prefix -- - cf in this example -- followed by a colon and then a - column qualifier suffix (a in this case). - + row1, column cf:a with a value of + value1. Columns in HBase are comprised of a column family prefix -- + cf in this example -- followed by a colon and then a + column qualifier suffix (a in this case). + Verify the data insert by running a scan of the table as follows - + hbase(main):007:0> scan 'test' ROW COLUMN+CELL row1 column=cf:a, timestamp=1288380727188, value=value1 row2 column=cf:b, timestamp=1288380738440, value=value2 row3 column=cf:c, timestamp=1288380747365, value=value3 3 row(s) in 0.0590 seconds - + Get a single row - + hbase(main):008:0> get 'test', 'row1' COLUMN CELL cf:a timestamp=1288380727188, value=value1 1 row(s) in 0.0400 seconds - + Now, disable and drop your table. This will clean up all done - above. - + above. + hbase(main):012:0> disable 'test' 0 row(s) in 1.0930 seconds hbase(main):013:0> drop 'test' 0 row(s) in 0.0770 seconds - + Exit the shell by typing exit. - + hbase(main):014:0> exit
- +
Stopping HBase - + Stop your hbase instance by running the stop script. - + $ ./bin/stop-hbase.sh stopping hbase...............
- +
Where to go next - + The above described standalone setup is good for testing and - experiments only. In the next chapter, , - we'll go into depth on the different HBase run modes, system requirements - running HBase, and critical configurations setting up a distributed HBase deploy. + experiments only. In the next chapter, , + we'll go into depth on the different HBase run modes, system requirements + running HBase, and critical configurations setting up a distributed HBase deploy.
- + diff --git a/src/main/docbkx/ops_mgt.xml b/src/main/docbkx/ops_mgt.xml index dbd6d17bf37..7b2cd63298f 100644 --- a/src/main/docbkx/ops_mgt.xml +++ b/src/main/docbkx/ops_mgt.xml @@ -1,13 +1,9 @@ - - Apache HBase Operational Management - This chapter will cover operational tools and practices required of a running Apache HBase cluster. - The subject of operations is related to the topics of , , - and but is a distinct topic in itself. + This chapter will cover operational tools and practices required of a running Apache HBase + cluster. The subject of operations is related to the topics of , , and but is a distinct topic in + itself.
- HBase Tools and Utilities + HBase Tools and Utilities Here we list HBase tools for administration, analysis, fixup, and debugging. -
Canary -There is a Canary class can help users to canary-test the HBase cluster status, with every - column-family for every regions or regionservers granularity. To see the usage, - $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -help - Will output - Usage: bin/hbase org.apache.hadoop.hbase.tool.Canary [opts] [table1 [table2]...] | [regionserver1 [regionserver2]..] +
+ Canary + There is a Canary class can help users to canary-test the HBase cluster status, with + every column-family for every regions or regionservers granularity. To see the usage, use + the --help parameter. + $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -help + +Usage: bin/hbase org.apache.hadoop.hbase.tool.Canary [opts] [table1 [table2]...] | [regionserver1 [regionserver2]..] where [opts] are: -help Show this help and exit. -regionserver replace the table argument to regionserver, @@ -50,51 +49,54 @@ -e Use region/regionserver as regular expression which means the region/regionserver is regular expression pattern -f <B> stop whole program if first error occurs, default is true - -t <N> timeout for a check, default is 600000 (milliseconds) - This tool will return non zero error codes to user for collaborating with other monitoring - tools, such as Nagios. The error code definitions are... - private static final int USAGE_EXIT_CODE = 1; + -t <N> timeout for a check, default is 600000 (milliseconds) + This tool will return non zero error codes to user for collaborating with other + monitoring tools, such as Nagios. The error code definitions are: + private static final int USAGE_EXIT_CODE = 1; private static final int INIT_ERROR_EXIT_CODE = 2; private static final int TIMEOUT_ERROR_EXIT_CODE = 3; private static final int ERROR_EXIT_CODE = 4; - Here are some examples based on the following given case. There are two HTable called + Here are some examples based on the following given case. There are two HTable called test-01 and test-02, they have two column family cf1 and cf2 respectively, and deployed on - the 3 regionservers. see following table. - - - - - - - RegionServer - test-01 - test-02 - - - - - rs1 - r1 - r2 - - - rs2 - r2 - - - - rs3 - r2 - r1 - - - -
Following are some examples based on the previous given case. -
Canary test for every column family (store) of every region of every table - -$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -The output log is... -13/12/09 03:26:32 INFO tool.Canary: read from region test-01,,1386230156732.0e3c7d77ffb6361ea1b996ac1042ca9a. column family cf1 in 2ms + the 3 regionservers. see following table. + + + + + + + + + RegionServer + test-01 + test-02 + + + + + rs1 + r1 + r2 + + + rs2 + r2 + + + + rs3 + r2 + r1 + + + + + Following are some examples based on the previous given case. +
+ Canary test for every column family (store) of every region of every table + $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary + +3/12/09 03:26:32 INFO tool.Canary: read from region test-01,,1386230156732.0e3c7d77ffb6361ea1b996ac1042ca9a. column family cf1 in 2ms 13/12/09 03:26:32 INFO tool.Canary: read from region test-01,,1386230156732.0e3c7d77ffb6361ea1b996ac1042ca9a. column family cf2 in 2ms 13/12/09 03:26:32 INFO tool.Canary: read from region test-01,0004883,1386230156732.87b55e03dfeade00f441125159f8ca87. column family cf1 in 4ms 13/12/09 03:26:32 INFO tool.Canary: read from region test-01,0004883,1386230156732.87b55e03dfeade00f441125159f8ca87. column family cf2 in 1ms @@ -104,64 +106,70 @@ The output log is... 13/12/09 03:26:32 INFO tool.Canary: read from region test-02,0004883,1386559511167.cbda32d5e2e276520712d84eaaa29d84. column family cf1 in 31ms 13/12/09 03:26:32 INFO tool.Canary: read from region test-02,0004883,1386559511167.cbda32d5e2e276520712d84eaaa29d84. column family cf2 in 8ms -So you can see, table test-01 has two regions and two column families, so the Canary tool will pick 4 small piece of data from 4 (2 region * 2 store) different stores. This is a default behavior of the this tool does. - -
+ So you can see, table test-01 has two regions and two column families, so the Canary + tool will pick 4 small piece of data from 4 (2 region * 2 store) different stores. This is + a default behavior of the this tool does. +
-
Canary test for every column family (store) of every region of specific table(s) - -You can also test one or more specific tables. -$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary test-01 test-02 - -
+
+ Canary test for every column family (store) of every region of specific + table(s) + You can also test one or more specific tables. + $ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary test-01 test-02 +
-
Canary test with regionserver granularity - -This will pick one small piece of data from each regionserver, and can also put your resionserver name as input options for canary-test specific regionservers. -$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -regionserver -The output log is... -13/12/09 06:05:17 INFO tool.Canary: Read from table:test-01 on region server:rs2 in 72ms +
+ Canary test with regionserver granularity + This will pick one small piece of data from each regionserver, and can also put your + resionserver name as input options for canary-test specific regionservers. + $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -regionserver + +13/12/09 06:05:17 INFO tool.Canary: Read from table:test-01 on region server:rs2 in 72ms 13/12/09 06:05:17 INFO tool.Canary: Read from table:test-02 on region server:rs3 in 34ms 13/12/09 06:05:17 INFO tool.Canary: Read from table:test-01 on region server:rs1 in 56ms - -
-
Canary test with regular expression pattern - -This will test both table test-01 and test-02. -$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -e test-0[1-2] - -
+
+
+ Canary test with regular expression pattern + This will test both table test-01 and test-02. + $ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -e test-0[1-2] +
-
Run canary test as daemon mode - -Run repeatedly with interval defined in option -interval whose default value is 6 seconds. This daemon will stop itself and return non-zero error code if any error occurs, due to the default value of option -f is true. -$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -daemon -Run repeatedly with internal 5 seconds and will not stop itself even error occurs in the test. -$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -daemon -interval 50000 -f false - -
+
+ Run canary test as daemon mode + Run repeatedly with interval defined in option -interval whose default value is 6 + seconds. This daemon will stop itself and return non-zero error code if any error occurs, + due to the default value of option -f is true. + $ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -daemon + Run repeatedly with internal 5 seconds and will not stop itself even error occurs in + the test. + $ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -daemon -interval 50000 -f false +
-
Force timeout if canary test stuck -In some cases, we suffered the request stucked on the regionserver and not response back to the client. The regionserver in problem, would also not indicated to be dead by Master, which would bring the clients hung. So we provide the timeout option to kill the canary test forcefully and return non-zero error code as well. -This run sets the timeout value to 60 seconds, the default value is 600 seconds. -$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -t 600000 - -
+
+ Force timeout if canary test stuck + In some cases, we suffered the request stucked on the regionserver and not response + back to the client. The regionserver in problem, would also not indicated to be dead by + Master, which would bring the clients hung. So we provide the timeout option to kill the + canary test forcefully and return non-zero error code as well. This run sets the timeout + value to 60 seconds, the default value is 600 seconds. + $ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -t 600000 +
-
Health Checker - You can configure HBase to run a script on a period and if it fails N times (configurable), have the server exit. - See HBASE-7351 Periodic health check script for configurations and detail. - +
+ Health Checker + You can configure HBase to run a script on a period and if it fails N times + (configurable), have the server exit. See HBASE-7351 Periodic health + check script for configurations and detail.
-
Driver - There is a Driver class that is executed by the HBase jar can be used to invoke frequently accessed utilities. For example, -HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar - -... will return... - +
+ Driver + There is a Driver class that is executed by the HBase jar can be used to + invoke frequently accessed utilities. For example, + HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar + An example program must be given as the first argument. Valid program names are: completebulkload: Complete a bulk data load. @@ -172,28 +180,23 @@ Valid program names are: rowcounter: Count rows in HBase table verifyrep: Compare the data from tables in two different clusters. WARNING: It doesn't work for incrementColumnValues'd cells since the timestamp is chan -... for allowable program names. -
- HBase <application>hbck</application> - An fsck for your HBase install - To run hbck against your HBase cluster run - $ ./bin/hbase hbck - At the end of the commands output it prints OK - or INCONSISTENCY. If your cluster reports - inconsistencies, pass -details to see more detail emitted. - If inconsistencies, run hbck a few times because the - inconsistency may be transient (e.g. cluster is starting up or a region is - splitting). - Passing -fix may correct the inconsistency (This latter - is an experimental feature). - - For more information, see . - + HBase <application>hbck</application> + An fsck for your HBase install + To run hbck against your HBase cluster run $ + ./bin/hbase hbck At the end of the command's output it prints + OK or INCONSISTENCY. If your cluster reports + inconsistencies, pass -details to see more detail emitted. If + inconsistencies, run hbck a few times because the inconsistency may be + transient (e.g. cluster is starting up or a region is splitting). Passing + -fix may correct the inconsistency (This latter is an experimental + feature). + For more information, see .
-
HFile Tool - See . +
+ HFile Tool + See .
WAL Tools @@ -201,109 +204,132 @@ Valid program names are:
<classname>FSHLog</classname> tool - The main method on FSHLog offers manual - split and dump facilities. Pass it WALs or the product of a split, the - content of the recovered.edits. directory. + The main method on FSHLog offers manual split and dump + facilities. Pass it WALs or the product of a split, the content of the + recovered.edits. directory. You can get a textual dump of a WAL file content by doing the - following: $ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.FSHLog --dump hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/10.10.21.10%3A60020.1283973724012 The - return code will be non-zero if issues with the file so you can test - wholesomeness of file by redirecting STDOUT to - /dev/null and testing the program return. + following: $ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.FSHLog --dump hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/10.10.21.10%3A60020.1283973724012 The + return code will be non-zero if issues with the file so you can test wholesomeness of file + by redirecting STDOUT to /dev/null and testing the program + return. Similarly you can force a split of a log file directory by - doing: $ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.FSHLog --split hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/ + doing: $ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.FSHLog --split hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/
<classname>HLogPrettyPrinter</classname> - HLogPrettyPrinter is a tool with configurable options to print the contents of an HLog. - + HLogPrettyPrinter is a tool with configurable options to + print the contents of an HLog.
-
Compression Tool - See . +
+ Compression Tool + See .
-
- CopyTable - - CopyTable is a utility that can copy part or of all of a table, either to the same cluster or another cluster. The target table must - first exist. The usage is as follows: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable [--starttime=X] [--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] tablename +
+ CopyTable + CopyTable is a utility that can copy part or of all of a table, either to the same + cluster or another cluster. The target table must first exist. The usage is as + follows: + $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable [--starttime=X] [--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] tablename - - - Options: - - starttime Beginning of the time range. Without endtime means starttime to forever. - endtime End of the time range. Without endtime means starttime to forever. - versions Number of cell versions to copy. - new.name New table's name. - peer.adr Address of the peer cluster given in the format hbase.zookeeper.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent - families Comma-separated list of ColumnFamilies to copy. - all.cells Also copy delete markers and uncollected deleted cells (advanced option). - - Args: - - tablename Name of table to copy. - - - Example of copying 'TestTable' to a cluster that uses replication for a 1 hour window: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable + + + Options + + starttime Beginning of the time range. Without endtime means + starttime to forever. + + + endtime End of the time range. Without endtime means starttime to + forever. + + + versions Number of cell versions to copy. + + + new.name New table's name. + + + peer.adr Address of the peer cluster given in the format + hbase.zookeeper.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent + + + families Comma-separated list of ColumnFamilies to copy. + + + all.cells Also copy delete markers and uncollected deleted cells + (advanced option). + + + + Args: + + tablename Name of table to copy. + + + Example of copying 'TestTable' to a cluster that uses replication for a 1 hour + window: + $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=1265875194289 --endtime=1265878794289 --peer.adr=server1,server2,server3:2181:/hbase TestTable - - Scanner Caching - Caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. - - - Versions - By default, CopyTable utility only copies the latest version of row cells unless --versions=n is explicitly specified in the command. - - - - See Jonathan Hsieh's Online HBase Backups with CopyTable blog post for more on CopyTable. - + + Scanner Caching + Caching for the input Scan is configured via hbase.client.scanner.caching + in the job configuration. + + + Versions + By default, CopyTable utility only copies the latest version of row cells unless + --versions=n is explicitly specified in the command. + + See Jonathan Hsieh's Online HBase Backups with CopyTable blog post for more on + CopyTable.
- Export - Export is a utility that will dump the contents of table to HDFS in a sequence file. Invoke via: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]] + Export + Export is a utility that will dump the contents of table to HDFS in a sequence file. + Invoke via: + $ bin/hbase org.apache.hadoop.hbase.mapreduce.Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]] - - Note: caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. - + + Note: caching for the input Scan is configured via + hbase.client.scanner.caching in the job configuration.
- Import - Import is a utility that will load data that has been exported back into HBase. Invoke via: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir> + Import + Import is a utility that will load data that has been exported back into HBase. Invoke + via: + $ bin/hbase org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir> - - To import 0.94 exported files in a 0.96 cluster or onwards, you need to set system property "hbase.import.version" when running the import command as below: -$ bin/hbase -Dhbase.import.version=0.94 org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir> + To import 0.94 exported files in a 0.96 cluster or onwards, you need to set system + property "hbase.import.version" when running the import command as below: + $ bin/hbase -Dhbase.import.version=0.94 org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir> -
- ImportTsv - ImportTsv is a utility that will load data in TSV format into HBase. It has two distinct usages: loading data from TSV format in HDFS - into HBase via Puts, and preparing StoreFiles to be loaded via the completebulkload. - - To load data via Puts (i.e., non-bulk loading): -$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c <tablename> <hdfs-inputdir> + ImportTsv + ImportTsv is a utility that will load data in TSV format into HBase. It has two distinct + usages: loading data from TSV format in HDFS into HBase via Puts, and preparing StoreFiles + to be loaded via the completebulkload. + To load data via Puts (i.e., non-bulk loading): + $ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c <tablename> <hdfs-inputdir> - - To generate StoreFiles for bulk-loading: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c -Dimporttsv.bulk.output=hdfs://storefile-outputdir <tablename> <hdfs-data-inputdir> + + To generate StoreFiles for bulk-loading: + $ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c -Dimporttsv.bulk.output=hdfs://storefile-outputdir <tablename> <hdfs-data-inputdir> - - These generated StoreFiles can be loaded into HBase via . - -
ImportTsv Options - Running ImportTsv with no arguments prints brief usage information: - + These generated StoreFiles can be loaded into HBase via . +
+ ImportTsv Options + Running ImportTsv with no arguments prints brief usage information: + Usage: importtsv -Dimporttsv.columns=a,b,c <tablename> <inputdir> Imports the given input directory of TSV data into the specified table. @@ -327,12 +353,13 @@ Other options that may be specified with -D include: -Dimporttsv.timestamp=currentTimeAsLong - use the specified timestamp for the import -Dimporttsv.mapper.class=my.Mapper - A user-defined Mapper to use instead of org.apache.hadoop.hbase.mapreduce.TsvImporterMapper -
-
ImportTsv Example - For example, assume that we are loading data into a table called 'datatsv' with a ColumnFamily called 'd' with two columns "c1" and "c2". - - Assume that an input file exists as follows: - +
+
+ ImportTsv Example + For example, assume that we are loading data into a table called 'datatsv' with a + ColumnFamily called 'd' with two columns "c1" and "c2". + Assume that an input file exists as follows: + row1 c1 c2 row2 c1 c2 row3 c1 c2 @@ -344,171 +371,187 @@ row8 c1 c2 row9 c1 c2 row10 c1 c2 - - For ImportTsv to use this imput file, the command line needs to look like this: - + + For ImportTsv to use this imput file, the command line needs to look like this: + HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2 -Dimporttsv.bulk.output=hdfs://storefileoutput datatsv hdfs://inputfile - ... and in this example the first column is the rowkey, which is why the HBASE_ROW_KEY is used. The second and third columns in the file will be imported as "d:c1" and "d:c2", respectively. - -
-
ImportTsv Warning - If you have preparing a lot of data for bulk loading, make sure the target HBase table is pre-split appropriately. - -
-
See Also - For more information about bulk-loading HFiles into HBase, see -
+ ... and in this example the first column is the rowkey, which is why the + HBASE_ROW_KEY is used. The second and third columns in the file will be imported as "d:c1" + and "d:c2", respectively. +
+
+ ImportTsv Warning + If you have preparing a lot of data for bulk loading, make sure the target HBase table + is pre-split appropriately. +
+
+ See Also + For more information about bulk-loading HFiles into HBase, see +
- CompleteBulkLoad - The completebulkload utility will move generated StoreFiles into an HBase table. This utility is often used - in conjunction with output from . - - There are two ways to invoke this utility, with explicit classname and via the driver: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles <hdfs://storefileoutput> <tablename> + CompleteBulkLoad + The completebulkload utility will move generated StoreFiles into an HBase + table. This utility is often used in conjunction with output from . + There are two ways to invoke this utility, with explicit classname and via the + driver: + $ bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles <hdfs://storefileoutput> <tablename> -.. and via the Driver.. -HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar completebulkload <hdfs://storefileoutput> <tablename> + .. and via the Driver.. + HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar completebulkload <hdfs://storefileoutput> <tablename> - -
CompleteBulkLoad Warning - Data generated via MapReduce is often created with file permissions that are not compatible with the running HBase process. Assuming you're running HDFS with permissions enabled, those permissions will need to be updated before you run CompleteBulkLoad. - -
- For more information about bulk-loading HFiles into HBase, see . - +
+ CompleteBulkLoad Warning + Data generated via MapReduce is often created with file permissions that are not + compatible with the running HBase process. Assuming you're running HDFS with permissions + enabled, those permissions will need to be updated before you run CompleteBulkLoad. + For more information about bulk-loading HFiles into HBase, see . +
+
- WALPlayer - WALPlayer is a utility to replay WAL files into HBase. - - The WAL can be replayed for a set of tables or all tables, and a - timerange can be provided (in milliseconds). The WAL is filtered to - this set of tables. The output can optionally be mapped to another set of tables. - - WALPlayer can also generate HFiles for later bulk importing, in that case - only a single table and no mapping can be specified. - - Invoke via: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer [options] <wal inputdir> <tables> [<tableMappings>]> + WALPlayer + WALPlayer is a utility to replay WAL files into HBase. + The WAL can be replayed for a set of tables or all tables, and a timerange can be + provided (in milliseconds). The WAL is filtered to this set of tables. The output can + optionally be mapped to another set of tables. + WALPlayer can also generate HFiles for later bulk importing, in that case only a single + table and no mapping can be specified. + Invoke via: + $ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer [options] <wal inputdir> <tables> [<tableMappings>]> - - For example: -$ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer /backuplogdir oldTable1,oldTable2 newTable1,newTable2 + For example: + $ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer /backuplogdir oldTable1,oldTable2 newTable1,newTable2 - - - WALPlayer, by default, runs as a mapreduce job. To NOT run WALPlayer as a mapreduce job on your cluster, - force it to run all in the local process by adding the flags -Dmapreduce.jobtracker.address=local on the command line. - + WALPlayer, by default, runs as a mapreduce job. To NOT run WALPlayer as a mapreduce job + on your cluster, force it to run all in the local process by adding the flags + -Dmapreduce.jobtracker.address=local on the command line.
- RowCounter and CellCounter - RowCounter is a - mapreduce job to count all the rows of a table. This is a good utility to use as a sanity check to ensure that HBase can read - all the blocks of a table if there are any concerns of metadata inconsistency. It will run the mapreduce all in a single - process but it will run faster if you have a MapReduce cluster in place for it to exploit. -$ bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter <tablename> [<column1> <column2>...] + RowCounter and CellCounter + RowCounter is a mapreduce job to count all the rows of a table. This is a good + utility to use as a sanity check to ensure that HBase can read all the blocks of a table if + there are any concerns of metadata inconsistency. It will run the mapreduce all in a single + process but it will run faster if you have a MapReduce cluster in place for it to + exploit. + $ bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter <tablename> [<column1> <column2>...] - - Note: caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. - - HBase ships another diagnostic mapreduce job called - CellCounter. Like - RowCounter, it gathers more fine-grained statistics about your table. The statistics gathered by RowCounter are more fine-grained - and include: - - Total number of rows in the table. - Total number of CFs across all rows. - Total qualifiers across all rows. - Total occurrence of each CF. - Total occurrence of each qualifier. - Total number of versions of each qualifier. - - - The program allows you to limit the scope of the run. Provide a row regex or prefix to limit the rows to analyze. Use - hbase.mapreduce.scan.column.family to specify scanning a single column family. - $ bin/hbase org.apache.hadoop.hbase.mapreduce.CellCounter <tablename> <outputDir> [regex or prefix] - - Note: just like RowCounter, caching for the input Scan is configured via hbase.client.scanner.caching in the - job configuration. + Note: caching for the input Scan is configured via + hbase.client.scanner.caching in the job configuration. + HBase ships another diagnostic mapreduce job called CellCounter. Like RowCounter, it gathers more fine-grained statistics about your + table. The statistics gathered by RowCounter are more fine-grained and include: + + + Total number of rows in the table. + + + Total number of CFs across all rows. + + + Total qualifiers across all rows. + + + Total occurrence of each CF. + + + Total occurrence of each qualifier. + + + Total number of versions of each qualifier. + + + The program allows you to limit the scope of the run. Provide a row regex or prefix to + limit the rows to analyze. Use hbase.mapreduce.scan.column.family to specify + scanning a single column family. + $ bin/hbase org.apache.hadoop.hbase.mapreduce.CellCounter <tablename> <outputDir> [regex or prefix] + Note: just like RowCounter, caching for the input Scan is configured via + hbase.client.scanner.caching in the job configuration.
- mlockall - It is possible to optionally pin your servers in physical memory making them less likely - to be swapped out in oversubscribed environments by having the servers call - mlockall on startup. - See HBASE-4391 Add ability to start RS as root and call mlockall - for how to build the optional library and have it run on startup. - + mlockall + It is possible to optionally pin your servers in physical memory making them less likely + to be swapped out in oversubscribed environments by having the servers call mlockall on startup. See HBASE-4391 Add ability to + start RS as root and call mlockall for how to build the optional library and have + it run on startup.
- Offline Compaction Tool - See the usage for the Compaction Tool. - Run it like this ./bin/hbase org.apache.hadoop.hbase.regionserver.CompactionTool - + Offline Compaction Tool + See the usage for the Compaction Tool. Run it like this ./bin/hbase + org.apache.hadoop.hbase.regionserver.CompactionTool +
-
+
+
Region Management
Major Compaction - Major compactions can be requested via the HBase shell or HBaseAdmin.majorCompact. - - Note: major compactions do NOT do region merges. See for more information about compactions. - - + Major compactions can be requested via the HBase shell or HBaseAdmin.majorCompact. + Note: major compactions do NOT do region merges. See for + more information about compactions.
Merge - Merge is a utility that can merge adjoining regions in the same table (see org.apache.hadoop.hbase.util.Merge). -$ bin/hbase org.apache.hadoop.hbase.util.Merge <tablename> <region1> <region2> + Merge is a utility that can merge adjoining regions in the same table (see + org.apache.hadoop.hbase.util.Merge). + $ bin/hbase org.apache.hadoop.hbase.util.Merge <tablename> <region1> <region2> - If you feel you have too many regions and want to consolidate them, Merge is the utility you need. Merge must - run be done when the cluster is down. - See the O'Reilly HBase Book for - an example of usage. - - You will need to pass 3 parameters to this application. The first one is the table name. The second one is the fully - qualified name of the first region to merge, like "table_name,\x0A,1342956111995.7cef47f192318ba7ccc75b1bbf27a82b.". The third one - is the fully qualified name for the second region to merge. - - Additionally, there is a Ruby script attached to HBASE-1621 - for region merging. - + If you feel you have too many regions and want to consolidate them, Merge is the utility + you need. Merge must run be done when the cluster is down. See the O'Reilly HBase + Book for an example of usage. + You will need to pass 3 parameters to this application. The first one is the table name. + The second one is the fully qualified name of the first region to merge, like + "table_name,\x0A,1342956111995.7cef47f192318ba7ccc75b1bbf27a82b.". The third one is the + fully qualified name for the second region to merge. + Additionally, there is a Ruby script attached to HBASE-1621 for region + merging.
-
Node Management -
Node Decommission - You can stop an individual RegionServer by running the following - script in the HBase directory on the particular node: - $ ./bin/hbase-daemon.sh stop regionserver - The RegionServer will first close all regions and then shut itself down. - On shutdown, the RegionServer's ephemeral node in ZooKeeper will expire. - The master will notice the RegionServer gone and will treat it as - a 'crashed' server; it will reassign the nodes the RegionServer was carrying. - Disable the Load Balancer before Decommissioning a node - If the load balancer runs while a node is shutting down, then - there could be contention between the Load Balancer and the - Master's recovery of the just decommissioned RegionServer. - Avoid any problems by disabling the balancer first. - See below. - - - - - A downside to the above stop of a RegionServer is that regions could be offline for - a good period of time. Regions are closed in order. If many regions on the server, the - first region to close may not be back online until all regions close and after the master - notices the RegionServer's znode gone. In Apache HBase 0.90.2, we added facility for having - a node gradually shed its load and then shutdown itself down. Apache HBase 0.90.2 added the - graceful_stop.sh script. Here is its usage: - $ ./bin/graceful_stop.sh +
+ Node Management +
+ Node Decommission + You can stop an individual RegionServer by running the following script in the HBase + directory on the particular node: + $ ./bin/hbase-daemon.sh stop regionserver + The RegionServer will first close all regions and then shut itself down. On shutdown, + the RegionServer's ephemeral node in ZooKeeper will expire. The master will notice the + RegionServer gone and will treat it as a 'crashed' server; it will reassign the nodes the + RegionServer was carrying. + + Disable the Load Balancer before Decommissioning a node + If the load balancer runs while a node is shutting down, then there could be + contention between the Load Balancer and the Master's recovery of the just decommissioned + RegionServer. Avoid any problems by disabling the balancer first. See + below. + + A downside to the above stop of a RegionServer is that regions could be offline for a + good period of time. Regions are closed in order. If many regions on the server, the first + region to close may not be back online until all regions close and after the master notices + the RegionServer's znode gone. In Apache HBase 0.90.2, we added facility for having a node + gradually shed its load and then shutdown itself down. Apache HBase 0.90.2 added the + graceful_stop.sh script. Here is its usage: + $ ./bin/graceful_stop.sh Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thrift] [--rest] &hostname> thrift If we should stop/start thrift before/after the hbase stop/start rest If we should stop/start rest before/after the hbase stop/start @@ -516,372 +559,430 @@ Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thri reload Move offloaded regions back on to the stopped server debug Move offloaded regions back on to the stopped server hostname Hostname of server we are to stop + To decommission a loaded RegionServer, run the following: $ + ./bin/graceful_stop.sh HOSTNAME where HOSTNAME is the host + carrying the RegionServer you would decommission. + + On <varname>HOSTNAME</varname> + The HOSTNAME passed to graceful_stop.sh must + match the hostname that hbase is using to identify RegionServers. Check the list of + RegionServers in the master UI for how HBase is referring to servers. Its usually hostname + but can also be FQDN. Whatever HBase is using, this is what you should pass the + graceful_stop.sh decommission script. If you pass IPs, the script + is not yet smart enough to make a hostname (or FQDN) of it and so it will fail when it + checks if server is currently running; the graceful unloading of regions will not run. - - To decommission a loaded RegionServer, run the following: - $ ./bin/graceful_stop.sh HOSTNAME - where HOSTNAME is the host carrying the RegionServer - you would decommission. - On <varname>HOSTNAME</varname> - The HOSTNAME passed to graceful_stop.sh - must match the hostname that hbase is using to identify RegionServers. - Check the list of RegionServers in the master UI for how HBase is - referring to servers. Its usually hostname but can also be FQDN. - Whatever HBase is using, this is what you should pass the - graceful_stop.sh decommission - script. If you pass IPs, the script is not yet smart enough to make - a hostname (or FQDN) of it and so it will fail when it checks if server is - currently running; the graceful unloading of regions will not run. - - The graceful_stop.sh script will move the regions off the - decommissioned RegionServer one at a time to minimize region churn. - It will verify the region deployed in the new location before it - will moves the next region and so on until the decommissioned server - is carrying zero regions. At this point, the graceful_stop.sh - tells the RegionServer stop. The master will at this point notice the - RegionServer gone but all regions will have already been redeployed - and because the RegionServer went down cleanly, there will be no - WAL logs to split. - Load Balancer - - It is assumed that the Region Load Balancer is disabled while the - graceful_stop script runs (otherwise the balancer - and the decommission script will end up fighting over region deployments). - Use the shell to disable the balancer: - hbase(main):001:0> balance_switch false + + The graceful_stop.sh script will move the regions off the + decommissioned RegionServer one at a time to minimize region churn. It will verify the + region deployed in the new location before it will moves the next region and so on until the + decommissioned server is carrying zero regions. At this point, the + graceful_stop.sh tells the RegionServer stop. The + master will at this point notice the RegionServer gone but all regions will have already + been redeployed and because the RegionServer went down cleanly, there will be no WAL logs to + split. + + Load Balancer + It is assumed that the Region Load Balancer is disabled while the + graceful_stop script runs (otherwise the balancer and the + decommission script will end up fighting over region deployments). Use the shell to + disable the balancer: + hbase(main):001:0> balance_switch false true 0 row(s) in 0.3590 seconds -This turns the balancer OFF. To reenable, do: - hbase(main):001:0> balance_switch true + This turns the balancer OFF. To reenable, do: + hbase(main):001:0> balance_switch true false 0 row(s) in 0.3590 seconds - - The graceful_stop will check the balancer - and if enabled, will turn it off before it goes to work. If it - exits prematurely because of error, it will not have reset the - balancer. Hence, it is better to manage the balancer apart from - graceful_stop reenabling it after you are done - w/ graceful_stop. - - + The graceful_stop will check the balancer and if enabled, will turn + it off before it goes to work. If it exits prematurely because of error, it will not have + reset the balancer. Hence, it is better to manage the balancer apart from + graceful_stop reenabling it after you are done w/ graceful_stop. -
- Decommissioning several Regions Servers concurrently - If you have a large cluster, you may want to - decommission more than one machine at a time by gracefully - stopping mutiple RegionServers concurrently. - To gracefully drain multiple regionservers at the - same time, RegionServers can be put into a "draining" - state. This is done by marking a RegionServer as a - draining node by creating an entry in ZooKeeper under the - hbase_root/draining znode. This znode has format - name,port,startcode just like the regionserver entries - under hbase_root/rs znode. - - Without this facility, decommissioning mulitple nodes - may be non-optimal because regions that are being drained - from one region server may be moved to other regionservers that - are also draining. Marking RegionServers to be in the - draining state prevents this from happeningSee - this blog - post for more details.. - -
+ +
+ Decommissioning several Regions Servers concurrently + If you have a large cluster, you may want to decommission more than one machine at a + time by gracefully stopping mutiple RegionServers concurrently. To gracefully drain + multiple regionservers at the same time, RegionServers can be put into a "draining" state. + This is done by marking a RegionServer as a draining node by creating an entry in + ZooKeeper under the hbase_root/draining znode. This znode has format + name,port,startcode just like the regionserver entries under + hbase_root/rs znode. + Without this facility, decommissioning mulitple nodes may be non-optimal because + regions that are being drained from one region server may be moved to other regionservers + that are also draining. Marking RegionServers to be in the draining state prevents this + from happening + See this blog post for more details. + . +
-
- Bad or Failing Disk - It is good having set if you have a decent number of disks - per machine for the case where a disk plain dies. But usually disks do the "John Wayne" -- i.e. take a while - to go down spewing errors in dmesg -- or for some reason, run much slower than their - companions. In this case you want to decommission the disk. You have two options. You can - decommission the datanode - or, less disruptive in that only the bad disks data will be rereplicated, can stop the datanode, - unmount the bad volume (You can't umount a volume while the datanode is using it), and then restart the - datanode (presuming you have set dfs.datanode.failed.volumes.tolerated > 0). The regionserver will - throw some errors in its logs as it recalibrates where to get its data from -- it will likely - roll its WAL log too -- but in general but for some latency spikes, it should keep on chugging. - - Short Circuit Reads - If you are doing short-circuit reads, you will have to move the regions off the regionserver - before you stop the datanode; when short-circuiting reading, though chmod'd so regionserver cannot - have access, because it already has the files open, it will be able to keep reading the file blocks - from the bad disk even though the datanode is down. Move the regions back after you restart the - datanode. - - -
-
-
- Rolling Restart - - You can also ask this script to restart a RegionServer after the shutdown - AND move its old regions back into place. The latter you might do to - retain data locality. A primitive rolling restart might be effected by - running something like the following: - $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i; done &> /tmp/log.txt & +
+ Bad or Failing Disk + It is good having set if you + have a decent number of disks per machine for the case where a disk plain dies. But + usually disks do the "John Wayne" -- i.e. take a while to go down spewing errors in + dmesg -- or for some reason, run much slower than their companions. + In this case you want to decommission the disk. You have two options. You can decommission the datanode or, less disruptive in that only the bad disks data + will be rereplicated, can stop the datanode, unmount the bad volume (You can't umount a + volume while the datanode is using it), and then restart the datanode (presuming you have + set dfs.datanode.failed.volumes.tolerated > 0). The regionserver will throw some errors in + its logs as it recalibrates where to get its data from -- it will likely roll its WAL log + too -- but in general but for some latency spikes, it should keep on chugging. + + Short Circuit Reads + If you are doing short-circuit reads, you will have to move the regions off the + regionserver before you stop the datanode; when short-circuiting reading, though chmod'd + so regionserver cannot have access, because it already has the files open, it will be + able to keep reading the file blocks from the bad disk even though the datanode is down. + Move the regions back after you restart the datanode. + +
+
+
+ Rolling Restart + You can also ask this script to restart a RegionServer after the shutdown AND move its + old regions back into place. The latter you might do to retain data locality. A primitive + rolling restart might be effected by running something like the following: + $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i; done &> /tmp/log.txt & + + Tail the output of /tmp/log.txt to follow the scripts progress. + The above does RegionServers only. The script will also disable the load balancer before + moving the regions. You'd need to do the master update separately. Do it before you run the + above script. Here is a pseudo-script for how you might craft a rolling restart script: + + + Untar your release, make sure of its configuration and then rsync it across the + cluster. If this is 0.90.2, patch it with HBASE-3744 and HBASE-3756. + + + Run hbck to ensure the cluster consistent + $ ./bin/hbase hbck Effect repairs if inconsistent. + + + + Restart the Master: + $ ./bin/hbase-daemon.sh stop master; ./bin/hbase-daemon.sh start master + + + + Run the graceful_stop.sh script per RegionServer. For + example: + $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i; done &> /tmp/log.txt & - Tail the output of /tmp/log.txt to follow the scripts - progress. The above does RegionServers only. The script will also disable the - load balancer before moving the regions. You'd need to do the master - update separately. Do it before you run the above script. - Here is a pseudo-script for how you might craft a rolling restart script: - - Untar your release, make sure of its configuration and - then rsync it across the cluster. If this is 0.90.2, patch it - with HBASE-3744 and HBASE-3756. - - - - Run hbck to ensure the cluster consistent - $ ./bin/hbase hbck - Effect repairs if inconsistent. - - - - Restart the Master: $ ./bin/hbase-daemon.sh stop master; ./bin/hbase-daemon.sh start master - - - - Run the graceful_stop.sh script per RegionServer. For example: - $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i; done &> /tmp/log.txt & - - If you are running thrift or rest servers on the RegionServer, pass --thrift or --rest options (See usage - for graceful_stop.sh script). - - - - Restart the Master again. This will clear out dead servers list and reenable the balancer. - - - - Run hbck to ensure the cluster is consistent. - - - - - It is important to drain HBase regions slowly when - restarting regionservers. Otherwise, multiple regions go - offline simultaneously as they are re-assigned to other - nodes. Depending on your usage patterns, this might not be - desirable. - + If you are running thrift or rest servers on the RegionServer, pass --thrift or + --rest options (See usage for graceful_stop.sh script). + + + Restart the Master again. This will clear out dead servers list and reenable the + balancer. + + + Run hbck to ensure the cluster is consistent. + + + It is important to drain HBase regions slowly when restarting regionservers. Otherwise, + multiple regions go offline simultaneously as they are re-assigned to other nodes. Depending + on your usage patterns, this might not be desirable.
- Adding a New Node - Adding a new regionserver in HBase is essentially free, you simply start it like this: - $ ./bin/hbase-daemon.sh start regionserver - and it will register itself with the master. Ideally you also started a DataNode on the same - machine so that the RS can eventually start to have local files. If you rely on ssh to start your - daemons, don't forget to add the new hostname in conf/regionservers on the master. - - At this point the region server isn't serving data because no regions have moved to it yet. If the balancer is - enabled, it will start moving regions to the new RS. On a small/medium cluster this can have a very adverse effect - on latency as a lot of regions will be offline at the same time. It is thus recommended to disable the balancer - the same way it's done when decommissioning a node and move the regions manually (or even better, using a script - that moves them one by one). - - The moved regions will all have 0% locality and won't have any blocks in cache so the region server will have - to use the network to serve requests. Apart from resulting in higher latency, it may also be able to use all of - your network card's capacity. For practical purposes, consider that a standard 1GigE NIC won't be able to read - much more than 100MB/s. In this case, or if you are in a OLAP environment and require having - locality, then it is recommended to major compact the moved regions. - + Adding a New Node + Adding a new regionserver in HBase is essentially free, you simply start it like this: + $ ./bin/hbase-daemon.sh start regionserver and it will register itself + with the master. Ideally you also started a DataNode on the same machine so that the RS can + eventually start to have local files. If you rely on ssh to start your daemons, don't forget + to add the new hostname in conf/regionservers on the master. + At this point the region server isn't serving data because no regions have moved to it + yet. If the balancer is enabled, it will start moving regions to the new RS. On a + small/medium cluster this can have a very adverse effect on latency as a lot of regions will + be offline at the same time. It is thus recommended to disable the balancer the same way + it's done when decommissioning a node and move the regions manually (or even better, using a + script that moves them one by one). + The moved regions will all have 0% locality and won't have any blocks in cache so the + region server will have to use the network to serve requests. Apart from resulting in higher + latency, it may also be able to use all of your network card's capacity. For practical + purposes, consider that a standard 1GigE NIC won't be able to read much more than + 100MB/s. In this case, or if you are in a OLAP environment and + require having locality, then it is recommended to major compact the moved regions.
-
+
+
- HBase Metrics -
- Metric Setup - See Metrics for - an introduction and how to enable Metrics emission. Still valid for HBase 0.94.x. - - For HBase 0.95.x and up, see - -
-
- Warning To Ganglia Users - Warning to Ganglia Users: by default, HBase will emit a LOT of metrics per RegionServer which may swamp your installation. - Options include either increasing Ganglia server capacity, or configuring HBase to emit fewer metrics. - -
-
- Most Important RegionServer Metrics -
<varname>blockCacheExpressCachingRatio (formerly blockCacheHitCachingRatio)</varname> - Block cache hit caching ratio (0 to 100). The cache-hit ratio for reads configured to look in the cache (i.e., cacheBlocks=true). -
-
<varname>callQueueLength</varname> - Point in time length of the RegionServer call queue. If requests arrive faster than the RegionServer handlers can process - them they will back up in the callQueue. -
-
<varname>compactionQueueLength (formerly compactionQueueSize)</varname> - Point in time length of the compaction queue. This is the number of Stores in the RegionServer that have been targeted for compaction. -
-
<varname>flushQueueSize</varname> - Point in time number of enqueued regions in the MemStore awaiting flush. -
-
<varname>hdfsBlocksLocalityIndex</varname> - Point in time percentage of HDFS blocks that are local to this RegionServer. The higher the better. -
-
<varname>memstoreSizeMB</varname> - Point in time sum of all the memstore sizes in this RegionServer (MB). Watch for this nearing or exceeding - the configured high-watermark for MemStore memory in the RegionServer. -
-
<varname>numberOfOnlineRegions</varname> - Point in time number of regions served by the RegionServer. This is an important metric to track for RegionServer-Region density. - -
-
<varname>readRequestsCount</varname> - Number of read requests for this RegionServer since startup. Note: this is a 32-bit integer and can roll. -
-
<varname>slowHLogAppendCount</varname> - Number of slow HLog append writes for this RegionServer since startup, where "slow" is > 1 second. This is - a good "canary" metric for HDFS. -
-
<varname>usedHeapMB</varname> - Point in time amount of memory used by the RegionServer (MB). -
-
<varname>writeRequestsCount</varname> - Number of write requests for this RegionServer since startup. Note: this is a 32-bit integer and can roll. -
+ HBase Metrics +
+ Metric Setup + See Metrics for an + introduction and how to enable Metrics emission. Still valid for HBase 0.94.x. + For HBase 0.95.x and up, see + +
+
+ Warning To Ganglia Users + Warning to Ganglia Users: by default, HBase will emit a LOT of metrics per RegionServer + which may swamp your installation. Options include either increasing Ganglia server + capacity, or configuring HBase to emit fewer metrics. +
+
+ Most Important RegionServer Metrics +
+ <varname>blockCacheExpressCachingRatio (formerly + blockCacheHitCachingRatio)</varname> + Block cache hit caching ratio (0 to 100). The cache-hit ratio for reads configured to + look in the cache (i.e., cacheBlocks=true). +
+
+ <varname>callQueueLength</varname> + Point in time length of the RegionServer call queue. If requests arrive faster than + the RegionServer handlers can process them they will back up in the callQueue. +
+
+ <varname>compactionQueueLength (formerly compactionQueueSize)</varname> + Point in time length of the compaction queue. This is the number of Stores in the + RegionServer that have been targeted for compaction. +
+
+ <varname>flushQueueSize</varname> + Point in time number of enqueued regions in the MemStore awaiting flush. +
+
+ <varname>hdfsBlocksLocalityIndex</varname> + Point in time percentage of HDFS blocks that are local to this RegionServer. The + higher the better. +
+
+ <varname>memstoreSizeMB</varname> + Point in time sum of all the memstore sizes in this RegionServer (MB). Watch for this + nearing or exceeding the configured high-watermark for MemStore memory in the + RegionServer. +
+
+ <varname>numberOfOnlineRegions</varname> + Point in time number of regions served by the RegionServer. This is an important + metric to track for RegionServer-Region density. +
+
+ <varname>readRequestsCount</varname> + Number of read requests for this RegionServer since startup. Note: this is a 32-bit + integer and can roll. +
+
+ <varname>slowHLogAppendCount</varname> + Number of slow HLog append writes for this RegionServer since startup, where "slow" is + > 1 second. This is a good "canary" metric for HDFS. +
+
+ <varname>usedHeapMB</varname> + Point in time amount of memory used by the RegionServer (MB). +
+
+ <varname>writeRequestsCount</varname> + Number of write requests for this RegionServer since startup. Note: this is a 32-bit + integer and can roll. +
-
-
- Other RegionServer Metrics -
<varname>blockCacheCount</varname> - Point in time block cache item count in memory. This is the number of blocks of StoreFiles (HFiles) in the cache. -
-
<varname>blockCacheEvictedCount</varname> - Number of blocks that had to be evicted from the block cache due to heap size constraints by RegionServer since startup. -
-
<varname>blockCacheFreeMB</varname> - Point in time block cache memory available (MB). -
-
<varname>blockCacheHitCount</varname> - Number of blocks of StoreFiles (HFiles) read from the cache by RegionServer since startup. -
-
<varname>blockCacheHitRatio</varname> - Block cache hit ratio (0 to 100) from RegionServer startup. Includes all read requests, although those with cacheBlocks=false - will always read from disk and be counted as a "cache miss", which means that full-scan MapReduce jobs can affect - this metric significantly. -
-
<varname>blockCacheMissCount</varname> - Number of blocks of StoreFiles (HFiles) requested but not read from the cache from RegionServer startup. -
-
<varname>blockCacheSizeMB</varname> - Point in time block cache size in memory (MB). i.e., memory in use by the BlockCache -
-
<varname>fsPreadLatency*</varname> - There are several filesystem positional read latency (ms) metrics, all measured from RegionServer startup. -
-
<varname>fsReadLatency*</varname> - There are several filesystem read latency (ms) metrics, all measured from RegionServer startup. The issue with - interpretation is that ALL reads go into this metric (e.g., single-record Gets, full table Scans), including - reads required for compactions. This metric is only interesting "over time" when comparing - major releases of HBase or your own code. -
-
<varname>fsWriteLatency*</varname> - There are several filesystem write latency (ms) metrics, all measured from RegionServer startup. The issue with - interpretation is that ALL writes go into this metric (e.g., single-record Puts, full table re-writes due to compaction). - This metric is only interesting "over time" when comparing - major releases of HBase or your own code. -
-
<varname>NumberOfStores</varname> - Point in time number of Stores open on the RegionServer. A Store corresponds to a ColumnFamily. For example, - if a table (which contains the column family) has 3 regions on a RegionServer, there will be 3 stores open for that - column family. -
-
<varname>NumberOfStorefiles</varname> - Point in time number of StoreFiles open on the RegionServer. A store may have more than one StoreFile (HFile). -
-
<varname>requestsPerSecond</varname> - Point in time number of read and write requests. Requests correspond to RegionServer RPC calls, - thus a single Get will result in 1 request, but a Scan with caching set to 1000 will result in 1 request for each 'next' call - (i.e., not each row). A bulk-load request will constitute 1 request per HFile. - This metric is less interesting than readRequestsCount and writeRequestsCount in terms of measuring activity - due to this metric being periodic. -
-
<varname>storeFileIndexSizeMB</varname> - Point in time sum of all the StoreFile index sizes in this RegionServer (MB) -
-
+
+
+ Other RegionServer Metrics +
+ <varname>blockCacheCount</varname> + Point in time block cache item count in memory. This is the number of blocks of + StoreFiles (HFiles) in the cache. +
+
+ <varname>blockCacheEvictedCount</varname> + Number of blocks that had to be evicted from the block cache due to heap size + constraints by RegionServer since startup. +
+
+ <varname>blockCacheFreeMB</varname> + Point in time block cache memory available (MB). +
+
+ <varname>blockCacheHitCount</varname> + Number of blocks of StoreFiles (HFiles) read from the cache by RegionServer since + startup. +
+
+ <varname>blockCacheHitRatio</varname> + Block cache hit ratio (0 to 100) from RegionServer startup. Includes all read + requests, although those with cacheBlocks=false will always read from disk and be counted + as a "cache miss", which means that full-scan MapReduce jobs can affect this metric + significantly. +
+
+ <varname>blockCacheMissCount</varname> + Number of blocks of StoreFiles (HFiles) requested but not read from the cache from + RegionServer startup. +
+
+ <varname>blockCacheSizeMB</varname> + Point in time block cache size in memory (MB). i.e., memory in use by the + BlockCache +
+
+ <varname>fsPreadLatency*</varname> + There are several filesystem positional read latency (ms) metrics, all measured from + RegionServer startup. +
+
+ <varname>fsReadLatency*</varname> + There are several filesystem read latency (ms) metrics, all measured from RegionServer + startup. The issue with interpretation is that ALL reads go into this metric (e.g., + single-record Gets, full table Scans), including reads required for compactions. This + metric is only interesting "over time" when comparing major releases of HBase or your own + code. +
+
+ <varname>fsWriteLatency*</varname> + There are several filesystem write latency (ms) metrics, all measured from + RegionServer startup. The issue with interpretation is that ALL writes go into this metric + (e.g., single-record Puts, full table re-writes due to compaction). This metric is only + interesting "over time" when comparing major releases of HBase or your own code. +
+
+ <varname>NumberOfStores</varname> + Point in time number of Stores open on the RegionServer. A Store corresponds to a + ColumnFamily. For example, if a table (which contains the column family) has 3 regions on + a RegionServer, there will be 3 stores open for that column family. +
+
+ <varname>NumberOfStorefiles</varname> + Point in time number of StoreFiles open on the RegionServer. A store may have more + than one StoreFile (HFile). +
+
+ <varname>requestsPerSecond</varname> + Point in time number of read and write requests. Requests correspond to RegionServer + RPC calls, thus a single Get will result in 1 request, but a Scan with caching set to 1000 + will result in 1 request for each 'next' call (i.e., not each row). A bulk-load request + will constitute 1 request per HFile. This metric is less interesting than + readRequestsCount and writeRequestsCount in terms of measuring activity due to this metric + being periodic. +
+
+ <varname>storeFileIndexSizeMB</varname> + Point in time sum of all the StoreFile index sizes in this RegionServer (MB) +
+
- HBase Monitoring + HBase Monitoring
- Overview - The following metrics are arguably the most important to monitor for each RegionServer for - "macro monitoring", preferably with a system like OpenTSDB. - If your cluster is having performance issues it's likely that you'll see something unusual with - this group. - - HBase: + Overview + The following metrics are arguably the most important to monitor for each RegionServer + for "macro monitoring", preferably with a system like OpenTSDB. If your cluster is having performance + issues it's likely that you'll see something unusual with this group. - See + HBase: + + See + - - OS: + - IO Wait - User CPU + OS: + + IO Wait + + + User CPU + - - Java: - GC + Java: + + GC + - - - - - For more information on HBase metrics, see . - + For more information on HBase metrics, see .
- Slow Query Log -The HBase slow query log consists of parseable JSON structures describing the properties of those client operations (Gets, Puts, Deletes, etc.) that either took too long to run, or produced too much output. The thresholds for "too long to run" and "too much output" are configurable, as described below. The output is produced inline in the main region server logs so that it is easy to discover further details from context with other logged events. It is also prepended with identifying tags (responseTooSlow), (responseTooLarge), (operationTooSlow), and (operationTooLarge) in order to enable easy filtering with grep, in case the user desires to see only slow queries. - + Slow Query Log + The HBase slow query log consists of parseable JSON structures describing the properties + of those client operations (Gets, Puts, Deletes, etc.) that either took too long to run, or + produced too much output. The thresholds for "too long to run" and "too much output" are + configurable, as described below. The output is produced inline in the main region server + logs so that it is easy to discover further details from context with other logged events. + It is also prepended with identifying tags (responseTooSlow), + (responseTooLarge), (operationTooSlow), and + (operationTooLarge) in order to enable easy filtering with grep, in + case the user desires to see only slow queries. -
Configuration -There are two configuration knobs that can be used to adjust the thresholds for when queries are logged. - +
+ Configuration + There are two configuration knobs that can be used to adjust the thresholds for when + queries are logged. - - -hbase.ipc.warn.response.time Maximum number of milliseconds that a query can be run without being logged. Defaults to 10000, or 10 seconds. Can be set to -1 to disable logging by time. - -hbase.ipc.warn.response.size Maximum byte size of response that a query can return without being logged. Defaults to 100 megabytes. Can be set to -1 to disable logging by size. - - -
+ + + hbase.ipc.warn.response.time Maximum number of milliseconds + that a query can be run without being logged. Defaults to 10000, or 10 seconds. Can be + set to -1 to disable logging by time. + + + hbase.ipc.warn.response.size Maximum byte size of response that + a query can return without being logged. Defaults to 100 megabytes. Can be set to -1 + to disable logging by size. + + +
-
Metrics -The slow query log exposes to metrics to JMX. -hadoop.regionserver_rpc_slowResponse a global metric reflecting the durations of all responses that triggered logging. -hadoop.regionserver_rpc_methodName.aboveOneSec A metric reflecting the durations of all responses that lasted for more than one second. - - -
+
+ Metrics + The slow query log exposes to metrics to JMX. + + + hadoop.regionserver_rpc_slowResponse a global metric reflecting + the durations of all responses that triggered logging. + + + hadoop.regionserver_rpc_methodName.aboveOneSec A metric + reflecting the durations of all responses that lasted for more than one second. + + -
Output -The output is tagged with operation e.g. (operationTooSlow) if the call was a client operation, such as a Put, Get, or Delete, which we expose detailed fingerprint information for. If not, it is tagged (responseTooSlow) and still produces parseable JSON output, but with less verbose information solely regarding its duration and size in the RPC itself. TooLarge is substituted for TooSlow if the response size triggered the logging, with TooLarge appearing even in the case that both size and duration triggered logging. - -
-
Example - -2011-09-08 10:01:25,824 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"tables":{"riley2":{"puts":[{"totalColumns":11,"families":{"actions":[{"timestamp":1315501284459,"qualifier":"0","vlen":9667580},{"timestamp":1315501284459,"qualifier":"1","vlen":10122412},{"timestamp":1315501284459,"qualifier":"2","vlen":11104617},{"timestamp":1315501284459,"qualifier":"3","vlen":13430635}]},"row":"cfcd208495d565ef66e7dff9f98764da:0"}],"families":["actions"]}},"processingtimems":956,"client":"10.47.34.63:33623","starttimems":1315501284456,"queuetimems":0,"totalPuts":1,"class":"HRegionServer","responsesize":0,"method":"multiPut"} - +
-Note that everything inside the "tables" structure is output produced by MultiPut's fingerprint, while the rest of the information is RPC-specific, such as processing time and client IP/port. Other client operations follow the same pattern and the same general structure, with necessary differences due to the nature of the individual operations. In the case that the call is not a client operation, that detailed fingerprint information will be completely absent. - +
+ Output + The output is tagged with operation e.g. (operationTooSlow) if + the call was a client operation, such as a Put, Get, or Delete, which we expose detailed + fingerprint information for. If not, it is tagged (responseTooSlow) + and still produces parseable JSON output, but with less verbose information solely + regarding its duration and size in the RPC itself. TooLarge is + substituted for TooSlow if the response size triggered the logging, + with TooLarge appearing even in the case that both size and duration + triggered logging. +
+
+ Example + + 2011-09-08 10:01:25,824 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"tables":{"riley2":{"puts":[{"totalColumns":11,"families":{"actions":[{"timestamp":1315501284459,"qualifier":"0","vlen":9667580},{"timestamp":1315501284459,"qualifier":"1","vlen":10122412},{"timestamp":1315501284459,"qualifier":"2","vlen":11104617},{"timestamp":1315501284459,"qualifier":"3","vlen":13430635}]},"row":"cfcd208495d565ef66e7dff9f98764da:0"}],"families":["actions"]}},"processingtimems":956,"client":"10.47.34.63:33623","starttimems":1315501284456,"queuetimems":0,"totalPuts":1,"class":"HRegionServer","responsesize":0,"method":"multiPut"} + -This particular example, for example, would indicate that the likely cause of slowness is simply a very large (on the order of 100MB) multiput, as we can tell by the "vlen," or value length, fields of each put in the multiPut. - -
-
+ Note that everything inside the "tables" structure is output produced by MultiPut's + fingerprint, while the rest of the information is RPC-specific, such as processing time + and client IP/port. Other client operations follow the same pattern and the same general + structure, with necessary differences due to the nature of the individual operations. In + the case that the call is not a client operation, that detailed fingerprint information + will be completely absent. + + This particular example, for example, would indicate that the likely cause of slowness + is simply a very large (on the order of 100MB) multiput, as we can tell by the "vlen," or + value length, fields of each put in the multiPut. +
+
@@ -889,252 +990,399 @@ false
Cluster Replication - See Cluster Replication. - + See Cluster + Replication.
- HBase Backup - There are two broad strategies for performing HBase backups: backing up with a full cluster shutdown, and backing up on a live cluster. - Each approach has pros and cons. - - For additional information, see HBase Backup Options over on the Sematext Blog. - -
Full Shutdown Backup - Some environments can tolerate a periodic full shutdown of their HBase cluster, for example if it is being used a back-end analytic capacity - and not serving front-end web-pages. The benefits are that the NameNode/Master are RegionServers are down, so there is no chance of missing - any in-flight changes to either StoreFiles or metadata. The obvious con is that the cluster is down. The steps include: - -
Stop HBase - - + HBase Backup + There are two broad strategies for performing HBase backups: backing up with a full + cluster shutdown, and backing up on a live cluster. Each approach has pros and cons. + For additional information, see HBase Backup + Options over on the Sematext Blog. +
+ Full Shutdown Backup + Some environments can tolerate a periodic full shutdown of their HBase cluster, for + example if it is being used a back-end analytic capacity and not serving front-end + web-pages. The benefits are that the NameNode/Master are RegionServers are down, so there is + no chance of missing any in-flight changes to either StoreFiles or metadata. The obvious con + is that the cluster is down. The steps include: +
+ Stop HBase +
-
Distcp - Distcp could be used to either copy the contents of the HBase directory in HDFS to either the same cluster in another directory, or - to a different cluster. - - Note: Distcp works in this situation because the cluster is down and there are no in-flight edits to files. - Distcp-ing of files in the HBase directory is not generally recommended on a live cluster. - +
+ Distcp + Distcp could be used to either copy the contents of the HBase directory in HDFS to + either the same cluster in another directory, or to a different cluster. + Note: Distcp works in this situation because the cluster is down and there are no + in-flight edits to files. Distcp-ing of files in the HBase directory is not generally + recommended on a live cluster.
-
Restore (if needed) - The backup of the hbase directory from HDFS is copied onto the 'real' hbase directory via distcp. The act of copying these files - creates new HDFS metadata, which is why a restore of the NameNode edits from the time of the HBase backup isn't required for this kind of - restore, because it's a restore (via distcp) of a specific HDFS directory (i.e., the HBase part) not the entire HDFS file-system. - +
+ Restore (if needed) + The backup of the hbase directory from HDFS is copied onto the 'real' hbase directory + via distcp. The act of copying these files creates new HDFS metadata, which is why a + restore of the NameNode edits from the time of the HBase backup isn't required for this + kind of restore, because it's a restore (via distcp) of a specific HDFS directory (i.e., + the HBase part) not the entire HDFS file-system.
-
Live Cluster Backup - Replication - This approach assumes that there is a second cluster. - See the HBase page on replication for more information. - +
+ Live Cluster Backup - Replication + This approach assumes that there is a second cluster. See the HBase page on replication for more + information.
-
Live Cluster Backup - CopyTable - The utility could either be used to copy data from one table to another on the - same cluster, or to copy data to another table on another cluster. - +
+ Live Cluster Backup - CopyTable + The utility could either be used to copy data from one table + to another on the same cluster, or to copy data to another table on another cluster. Since the cluster is up, there is a risk that edits could be missed in the copy process.
-
Live Cluster Backup - Export - The approach dumps the content of a table to HDFS on the same cluster. To restore the data, the - utility would be used. - - Since the cluster is up, there is a risk that edits could be missed in the export process. - +
+ Live Cluster Backup - Export + The approach dumps the content of a table to HDFS on the same + cluster. To restore the data, the utility would be used. + Since the cluster is up, there is a risk that edits could be missed in the export + process.
-
+
+
HBase Snapshots - HBase Snapshots allow you to take a snapshot of a table without too much impact on Region Servers. - Snapshot, Clone and restore operations don't involve data copying. - Also, Exporting the snapshot to another cluster doesn't have impact on the Region Servers. - - Prior to version 0.94.6, the only way to backup or to clone a table is to use CopyTable/ExportTable, - or to copy all the hfiles in HDFS after disabling the table. - The disadvantages of these methods are that you can degrade region server performance - (Copy/Export Table) or you need to disable the table, that means no reads or writes; - and this is usually unacceptable. - -
Configuration - To turn on the snapshot support just set the - hbase.snapshot.enabled property to true. - (Snapshots are enabled by default in 0.95+ and off by default in 0.94.6+) - + HBase Snapshots allow you to take a snapshot of a table without too much impact on Region + Servers. Snapshot, Clone and restore operations don't involve data copying. Also, Exporting + the snapshot to another cluster doesn't have impact on the Region Servers. + Prior to version 0.94.6, the only way to backup or to clone a table is to use + CopyTable/ExportTable, or to copy all the hfiles in HDFS after disabling the table. The + disadvantages of these methods are that you can degrade region server performance (Copy/Export + Table) or you need to disable the table, that means no reads or writes; and this is usually + unacceptable. +
+ Configuration + To turn on the snapshot support just set the hbase.snapshot.enabled + property to true. (Snapshots are enabled by default in 0.95+ and off by default in + 0.94.6+) + <property> <name>hbase.snapshot.enabled</name> <value>true</value> </property> -
-
Take a Snapshot - You can take a snapshot of a table regardless of whether it is enabled or disabled. - The snapshot operation doesn't involve any data copying. - - $ ./bin/hbase shell - hbase> snapshot 'myTable', 'myTableSnapshot-122112' - - +
+ Take a Snapshot + You can take a snapshot of a table regardless of whether it is enabled or disabled. The + snapshot operation doesn't involve any data copying. + +$ ./bin/hbase shell +hbase> snapshot 'myTable', 'myTableSnapshot-122112' +
-
Listing Snapshots - List all snapshots taken (by printing the names and relative information). - - $ ./bin/hbase shell - hbase> list_snapshots - - +
+ Listing Snapshots + List all snapshots taken (by printing the names and relative information). + +$ ./bin/hbase shell +hbase> list_snapshots +
-
Deleting Snapshots - You can remove a snapshot, and the files retained for that snapshot will be removed - if no longer needed. - - $ ./bin/hbase shell - hbase> delete_snapshot 'myTableSnapshot-122112' - - +
+ Deleting Snapshots + You can remove a snapshot, and the files retained for that snapshot will be removed if + no longer needed. + +$ ./bin/hbase shell +hbase> delete_snapshot 'myTableSnapshot-122112' +
-
Clone a table from snapshot - From a snapshot you can create a new table (clone operation) with the same data - that you had when the snapshot was taken. - The clone operation, doesn't involve data copies, and a change to the cloned table - doesn't impact the snapshot or the original table. - - $ ./bin/hbase shell - hbase> clone_snapshot 'myTableSnapshot-122112', 'myNewTestTable' - - +
+ Clone a table from snapshot + From a snapshot you can create a new table (clone operation) with the same data that you + had when the snapshot was taken. The clone operation, doesn't involve data copies, and a + change to the cloned table doesn't impact the snapshot or the original table. + +$ ./bin/hbase shell +hbase> clone_snapshot 'myTableSnapshot-122112', 'myNewTestTable' +
-
Restore a snapshot - The restore operation requires the table to be disabled, and the table will be - restored to the state at the time when the snapshot was taken, - changing both data and schema if required. - - $ ./bin/hbase shell - hbase> disable 'myTable' - hbase> restore_snapshot 'myTableSnapshot-122112' - - +
+ Restore a snapshot + The restore operation requires the table to be disabled, and the table will be restored + to the state at the time when the snapshot was taken, changing both data and schema if + required. + +$ ./bin/hbase shell +hbase> disable 'myTable' +hbase> restore_snapshot 'myTableSnapshot-122112' + - Since Replication works at log level and snapshots at file-system level, - after a restore, the replicas will be in a different state from the master. - If you want to use restore, you need to stop replication and redo the bootstrap. - + Since Replication works at log level and snapshots at file-system level, after a + restore, the replicas will be in a different state from the master. If you want to use + restore, you need to stop replication and redo the bootstrap. - In case of partial data-loss due to misbehaving client, instead of a full restore - that requires the table to be disabled, you can clone the table from the snapshot - and use a Map-Reduce job to copy the data that you need, from the clone to the main one. - + In case of partial data-loss due to misbehaving client, instead of a full restore that + requires the table to be disabled, you can clone the table from the snapshot and use a + Map-Reduce job to copy the data that you need, from the clone to the main one.
-
Snapshots operations and ACLs - If you are using security with the AccessController Coprocessor (See ), - only a global administrator can take, clone, or restore a snapshot, and these actions do not capture the ACL rights. - This means that restoring a table preserves the ACL rights of the existing table, - while cloning a table creates a new table that has no ACL rights until the administrator adds them. +
+ Snapshots operations and ACLs + If you are using security with the AccessController Coprocessor (See ), only a global administrator can take, + clone, or restore a snapshot, and these actions do not capture the ACL rights. This means + that restoring a table preserves the ACL rights of the existing table, while cloning a table + creates a new table that has no ACL rights until the administrator adds them.
-
Export to another cluster - The ExportSnapshot tool copies all the data related to a snapshot (hfiles, logs, snapshot metadata) to another cluster. - The tool executes a Map-Reduce job, similar to distcp, to copy files between the two clusters, - and since it works at file-system level the hbase cluster does not have to be online. - To copy a snapshot called MySnapshot to an HBase cluster srv2 (hdfs:///srv2:8082/hbase) using 16 mappers: -$ bin/hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs://srv2:8082/hbase -mappers 16 - - +
+ Export to another cluster + The ExportSnapshot tool copies all the data related to a snapshot (hfiles, logs, + snapshot metadata) to another cluster. The tool executes a Map-Reduce job, similar to + distcp, to copy files between the two clusters, and since it works at file-system level the + hbase cluster does not have to be online. + To copy a snapshot called MySnapshot to an HBase cluster srv2 (hdfs:///srv2:8082/hbase) + using 16 mappers: + $ bin/hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs://srv2:8082/hbase -mappers 16
-
+
+ -
Capacity Planning and Region Sizing - There are several considerations when planning the capacity for an HBase cluster and performing the initial configuration. Start with a solid understanding of how HBase handles data internally. -
Node count and hardware/VM configuration -
Physical data size -Physical data size on disk is distinct from logical size of your data and is affected by the following: - -Increased by HBase overhead - -See and . At least 24 bytes per key-value (cell), can be more. Small keys/values means more relative overhead. -KeyValue instances are aggregated into blocks, which are indexed. Indexes also have to be stored. Blocksize is configurable on a per-ColumnFamily basis. See . - -Decreased by and data block encoding, depending on data. See also this thread. You might want to test what compression and encoding (if any) make sense for your data. -Increased by size of region server (usually fixed and negligible - less than half of RS memory size, per RS). -Increased by HDFS replication - usually x3. - -Aside from the disk space necessary to store the data, one RS may not be able to serve arbitrarily large amounts of data due to some practical limits on region count and size (see ). -
-
Read/Write throughput -Number of nodes can also be driven by required thoughput for reads and/or writes. The throughput one can get per node depends a lot on data (esp. key/value sizes) and request patterns, as well as node and system configuration. Planning should be done for peak load if it is likely that the load would be the main driver of the increase of the node count. PerformanceEvaluation and tools can be used to test single node or a test cluster. -For write, usually 5-15Mb/s per RS can be expected, since every region server has only one active WAL. There's no good estimate for reads, as it depends vastly on data, requests, and cache hit rate. might be helpful. -
-
JVM GC limitations -RS cannot currently utilize very large heap due to cost of GC. There's also no good way of running multiple RS-es per server (other than running several VMs per machine). Thus, ~20-24Gb or less memory dedicated to one RS is recommended. GC tuning is required for large heap sizes. See , and elsewhere (TODO: where?) -
-
-
Determining region count and size -Generally less regions makes for a smoother running cluster (you can always manually split the big regions later (if necessary) to spread the data, or request load, over the cluster); 20-200 regions per RS is a reasonable range. The number of regions cannot be configured directly (unless you go for fully ); adjust the region size to achieve the target region size given table size. -When configuring regions for multiple tables, note that most region settings can be set on a per-table basis via HTableDescriptor, as well as shell commands. These settings will override the ones in hbase-site.xml. That is useful if your tables have different workloads/use cases. -Also note that in the discussion of region sizes here, HDFS replication factor is not (and should not be) taken into account, whereas other factors should be. So, if your data is compressed and replicated 3 ways by HDFS, "9 Gb region" means 9 Gb of compressed data. HDFS replication factor only affects your disk usage and is invisible to most HBase code. -
Number of regions per RS - upper bound -In production scenarios, where you have a lot of data, you are normally concerned with the maximum number of regions you can have per server. has technical discussion on the subject; in short, maximum number of regions is mostly determined by memstore memory usage. Each region has its own memstores; these grow up to a configurable size; usually in 128-256Mb range, see . There's one memstore per column family (so there's only one per region if there's one CF in the table). RS dedicates some fraction of total memory (see ) to region memstores. If this memory is exceeded (too much memstore usage), undesirable consequences such as unresponsive server, or later compaction storms, can result. Thus, a good starting point for the number of regions per RS (assuming one table) is (RS memory)*(total memstore fraction)/((memstore size)*(# column families)) -E.g. if RS has 16Gb RAM, with default settings, it is 16384*0.4/128 ~ 51 regions per RS is a starting point. The formula can be extended to multiple tables; if they all have the same configuration, just use total number of families. -This number can be adjusted; the formula above assumes all your regions are filled at approximately the same rate. If only a fraction of your regions are going to be actively written to, you can divide the result by that fraction to get a larger region count. Then, even if all regions are written to, all region memstores are not filled evenly, and eventually jitter appears even if they are (due to limited number of concurrent flushes). Thus, one can have as many as 2-3 times more regions than the starting point; however, increased numbers carry increased risk. -For write-heavy workload, memstore fraction can be increased in configuration at the expense of block cache; this will also allow one to have more regions. -
-
Number of regions per RS - lower bound -HBase scales by having regions across many servers. Thus if you have 2 regions for 16GB data, on a 20 node machine your data will be concentrated on just a few machines - nearly the entire cluster will be idle. This really can't be stressed enough, since a common problem is loading 200MB data into HBase and then wondering why your awesome 10 node cluster isn't doing anything. -On the other hand, if you have a very large amount of data, you may also want to go for a larger number of regions to avoid having regions that are too large. -
-
Maximum region size -For large tables in production scenarios, maximum region size is mostly limited by compactions - very large compactions, esp. major, can degrade cluster performance. Currently, the recommended maximum region size is 10-20Gb, and 5-10Gb is optimal. For older 0.90.x codebase, the upper-bound of regionsize is about 4Gb, with a default of 256Mb. -The size at which the region is split into two is generally configured via ; for details, see . -If you cannot estimate the size of your tables well, when starting off, it's probably best to stick to the default region size, perhaps going smaller for hot tables (or manually split hot regions to spread the load over the cluster), or go with larger region sizes if your cell sizes tend to be largish (100k and up). -In HBase 0.98, experimental stripe compactions feature was added that would allow for larger regions, especially for log data. See . -
-
Total data size per region server -According to above numbers for region size and number of regions per region server, in an optimistic estimate 10 GB x 100 regions per RS will give up to 1TB served per region server, which is in line with some of the reported multi-PB use cases. However, it is important to think about the data vs cache size ratio at the RS level. With 1TB of data per server and 10 GB block cache, only 1% of the data will be cached, which may barely cover all block indices. -
-
-
Initial configuration and tuning -First, see . Note that some configurations, more than others, depend on specific scenarios. Pay special attention to - - - request handler thread count, vital for high-throughput workloads. - - the blocking number of WAL files depends on your memstore configuration and should be set accordingly to prevent potential blocking when doing high volume of writes. - -Then, there are some considerations when setting up your cluster and tables. -
Compactions -Depending on read/write volume and latency requirements, optimal compaction settings may be different. See for some details. -When provisioning for large data sizes, however, it's good to keep in mind that compactions can affect write throughput. Thus, for write-intensive workloads, you may opt for less frequent compactions and more store files per regions. Minimum number of files for compactions (hbase.hstore.compaction.min) can be set to higher value; should also be increased, as more files might accumulate in such case. You may also consider manually managing compactions: -
-
Pre-splitting the table -Based on the target number of the regions per RS (see ) and number of RSes, one can pre-split the table at creation time. This would both avoid some costly splitting as the table starts to fill up, and ensure that the table starts out already distributed across many servers. -If the table is expected to grow large enough to justify that, at least one region per RS should be created. It is not recommended to split immediately into the full target number of regions (e.g. 50 * number of RSes), but a low intermediate value can be chosen. For multiple tables, it is recommended to be conservative with presplitting (e.g. pre-split 1 region per RS at most), especially if you don't know how much each table will grow. If you split too much, you may end up with too many regions, with some tables having too many small regions. -For pre-splitting howto, see . -
-
-
-
Table Rename - In versions 0.90.x of hbase and earlier, we had a simple script that would rename the hdfs - table directory and then do an edit of the .META. table replacing all mentions of the old - table name with the new. The script was called ./bin/rename_table.rb. - The script was deprecated and removed mostly because it was unmaintained and the operation - performed by the script was brutal. - - - As of hbase 0.94.x, you can use the snapshot facility renaming a table. Here is how you would -do it using the hbase shell: -hbase shell> disable 'tableName' +
+ Capacity Planning and Region Sizing + There are several considerations when planning the capacity for an HBase cluster and + performing the initial configuration. Start with a solid understanding of how HBase handles + data internally. +
+ Node count and hardware/VM configuration +
+ Physical data size + Physical data size on disk is distinct from logical size of your data and is affected + by the following: + + + Increased by HBase overhead + + + See and . At least 24 + bytes per key-value (cell), can be more. Small keys/values means more relative + overhead. + + + KeyValue instances are aggregated into blocks, which are indexed. Indexes also + have to be stored. Blocksize is configurable on a per-ColumnFamily basis. See + . + + + + + Decreased by and + data block encoding, depending on data. See also this thread. You might + want to test what compression and encoding (if any) make sense for your data. + + + Increased by size of region server + (usually fixed and negligible - less than half of RS memory size, per RS). + + + Increased by HDFS replication - usually x3. + + + Aside from the disk space necessary to store the data, one RS may not be able to serve + arbitrarily large amounts of data due to some practical limits on region count and size + (see ). +
+ +
+ Read/Write throughput + Number of nodes can also be driven by required thoughput for reads and/or writes. The + throughput one can get per node depends a lot on data (esp. key/value sizes) and request + patterns, as well as node and system configuration. Planning should be done for peak load + if it is likely that the load would be the main driver of the increase of the node count. + PerformanceEvaluation and tools can be + used to test single node or a test cluster. + For write, usually 5-15Mb/s per RS can be expected, since every region server has only + one active WAL. There's no good estimate for reads, as it depends vastly on data, + requests, and cache hit rate. might be helpful. +
+ +
+ JVM GC limitations + RS cannot currently utilize very large heap due to cost of GC. There's also no good + way of running multiple RS-es per server (other than running several VMs per machine). + Thus, ~20-24Gb or less memory dedicated to one RS is recommended. GC tuning is required + for large heap sizes. See , and + elsewhere (TODO: where?) +
+ +
+ +
+ Determining region count and size + Generally less regions makes for a smoother running cluster (you can always manually + split the big regions later (if necessary) to spread the data, or request load, over the + cluster); 20-200 regions per RS is a reasonable range. The number of regions cannot be + configured directly (unless you go for fully ); adjust the region size to achieve the target + region size given table size. + When configuring regions for multiple tables, note that most region settings can be set + on a per-table basis via HTableDescriptor, as well as shell commands. These settings will override the ones + in hbase-site.xml. That is useful if your tables have different + workloads/use cases. + Also note that in the discussion of region sizes here, HDFS + replication factor is not (and should not be) taken into account, whereas other factors + should + be. So, if your data is compressed and replicated 3 ways by HDFS, "9 Gb region" + means 9 Gb of compressed data. HDFS replication factor only affects your disk usage and is + invisible to most HBase code. +
+ Number of regions per RS - upper bound + In production scenarios, where you have a lot of data, you are normally concerned with + the maximum number of regions you can have per server. + has technical discussion on the subject; in short, maximum number of regions is mostly + determined by memstore memory usage. Each region has its own memstores; these grow up to a + configurable size; usually in 128-256Mb range, see . There's one memstore per column family + (so there's only one per region if there's one CF in the table). RS dedicates some + fraction of total memory (see ) + to region memstores. If this memory is exceeded (too much memstore usage), undesirable + consequences such as unresponsive server, or later compaction storms, can result. Thus, a + good starting point for the number of regions per RS (assuming one table) is: + + (RS memory)*(total memstore fraction)/((memstore size)*(# column families)) + E.g. if RS has 16Gb RAM, with default settings, it is 16384*0.4/128 ~ 51 regions per + RS is a starting point. The formula can be extended to multiple tables; if they all have + the same configuration, just use total number of families. + This number can be adjusted; the formula above assumes all your regions are filled at + approximately the same rate. If only a fraction of your regions are going to be actively + written to, you can divide the result by that fraction to get a larger region count. Then, + even if all regions are written to, all region memstores are not filled evenly, and + eventually jitter appears even if they are (due to limited number of concurrent flushes). + Thus, one can have as many as 2-3 times more regions than the starting point; however, + increased numbers carry increased risk. + For write-heavy workload, memstore fraction can be increased in configuration at the + expense of block cache; this will also allow one to have more regions. +
+ +
+ Number of regions per RS - lower bound + HBase scales by having regions across many servers. Thus if you have 2 regions for + 16GB data, on a 20 node machine your data will be concentrated on just a few machines - + nearly the entire cluster will be idle. This really can't be stressed enough, since a + common problem is loading 200MB data into HBase and then wondering why your awesome 10 + node cluster isn't doing anything. + On the other hand, if you have a very large amount of data, you may also want to go + for a larger number of regions to avoid having regions that are too large. +
+ +
+ Maximum region size + For large tables in production scenarios, maximum region size is mostly limited by + compactions - very large compactions, esp. major, can degrade cluster performance. + Currently, the recommended maximum region size is 10-20Gb, and 5-10Gb is optimal. For + older 0.90.x codebase, the upper-bound of regionsize is about 4Gb, with a default of + 256Mb. + The size at which the region is split into two is generally configured via ; for details, see . + If you cannot estimate the size of your tables well, when starting off, it's probably + best to stick to the default region size, perhaps going smaller for hot tables (or + manually split hot regions to spread the load over the cluster), or go with larger region + sizes if your cell sizes tend to be largish (100k and up). + In HBase 0.98, experimental stripe compactions feature was added that would allow for + larger regions, especially for log data. See . +
+ +
+ Total data size per region server + According to above numbers for region size and number of regions per region server, in + an optimistic estimate 10 GB x 100 regions per RS will give up to 1TB served per region + server, which is in line with some of the reported multi-PB use cases. However, it is + important to think about the data vs cache size ratio at the RS level. With 1TB of data + per server and 10 GB block cache, only 1% of the data will be cached, which may barely + cover all block indices. +
+ +
+ +
+ Initial configuration and tuning + First, see . Note that some configurations, + more than others, depend on specific scenarios. Pay special attention to: + + + - request handler thread count, + vital for high-throughput workloads. + + + - the blocking number of WAL files depends on your + memstore configuration and should be set accordingly to prevent potential blocking when + doing high volume of writes. + + + Then, there are some considerations when setting up your cluster and tables. +
+ Compactions + Depending on read/write volume and latency requirements, optimal compaction settings + may be different. See for some details. + When provisioning for large data sizes, however, it's good to keep in mind that + compactions can affect write throughput. Thus, for write-intensive workloads, you may opt + for less frequent compactions and more store files per regions. Minimum number of files + for compactions (hbase.hstore.compaction.min) can be set to higher + value; should also be increased, as more + files might accumulate in such case. You may also consider manually managing compactions: + +
+ +
+ Pre-splitting the table + Based on the target number of the regions per RS (see ) and number of RSes, + one can pre-split the table at creation time. This would both avoid some costly splitting + as the table starts to fill up, and ensure that the table starts out already distributed + across many servers. + If the table is expected to grow large enough to justify that, at least one region per + RS should be created. It is not recommended to split immediately into the full target + number of regions (e.g. 50 * number of RSes), but a low intermediate value can be chosen. + For multiple tables, it is recommended to be conservative with presplitting (e.g. + pre-split 1 region per RS at most), especially if you don't know how much each table will + grow. If you split too much, you may end up with too many regions, with some tables having + too many small regions. + For pre-splitting howto, see . +
+ +
+ +
+ +
+ Table Rename + In versions 0.90.x of hbase and earlier, we had a simple script that would rename the hdfs + table directory and then do an edit of the .META. table replacing all mentions of the old + table name with the new. The script was called ./bin/rename_table.rb. The + script was deprecated and removed mostly because it was unmaintained and the operation + performed by the script was brutal. + As of hbase 0.94.x, you can use the snapshot facility renaming a table. Here is how you + would do it using the hbase shell: + hbase shell> disable 'tableName' hbase shell> snapshot 'tableName', 'tableSnapshot' hbase shell> clone_snapshot 'tableSnapshot', 'newTableName' hbase shell> delete_snapshot 'tableSnapshot' hbase shell> drop 'tableName' -or in code it would be as follows: -void rename(HBaseAdmin admin, String oldTableName, String newTableName) { - String snapshotName = randomName(); - admin.disableTable(oldTableName); - admin.snapshot(snapshotName, oldTableName); - admin.cloneSnapshot(snapshotName, newTableName); - admin.deleteSnapshot(snapshotName); - admin.deleteTable(oldTableName); + or in code it would be as follows: + void rename(HBaseAdmin admin, String oldTableName, String newTableName) { +String snapshotName = randomName(); +admin.disableTable(oldTableName); +admin.snapshot(snapshotName, oldTableName); +admin.cloneSnapshot(snapshotName, newTableName); +admin.deleteSnapshot(snapshotName); +admin.deleteTable(oldTableName); } - +
diff --git a/src/main/docbkx/performance.xml b/src/main/docbkx/performance.xml index 854e7c9fa1d..a56c52293f8 100644 --- a/src/main/docbkx/performance.xml +++ b/src/main/docbkx/performance.xml @@ -51,9 +51,9 @@ Important items to consider: - Switching capacity of the device - Number of systems connected - Uplink capacity + Switching capacity of the device + Number of systems connected + Uplink capacity
@@ -71,9 +71,9 @@ Mitigation of this issue is fairly simple and can be accomplished in multiple ways: - Use appropriate hardware for the scale of the cluster which you're attempting to build. - Use larger single switch configurations i.e. single 48 port as opposed to 2x 24 port - Configure port trunking for uplinks to utilize multiple interfaces to increase cross switch bandwidth. + Use appropriate hardware for the scale of the cluster which you're attempting to build. + Use larger single switch configurations i.e. single 48 port as opposed to 2x 24 port + Configure port trunking for uplinks to utilize multiple interfaces to increase cross switch bandwidth.
@@ -81,8 +81,8 @@ Multiple Racks Multiple rack configurations carry the same potential issues as multiple switches, and can suffer performance degradation from two main areas: - Poor switch capacity performance - Insufficient uplink to another rack + Poor switch capacity performance + Insufficient uplink to another rack If the the switches in your rack have appropriate switching capacity to handle all the hosts at full speed, the next most likely issue will be caused by homing more of your cluster across racks. The easiest way to avoid issues when spanning multiple racks is to use port trunking to create a bonded uplink to other racks. diff --git a/src/main/docbkx/schema_design.xml b/src/main/docbkx/schema_design.xml index 482d3d12a4c..a79f1754975 100644 --- a/src/main/docbkx/schema_design.xml +++ b/src/main/docbkx/schema_design.xml @@ -32,7 +32,7 @@ the various non-rdbms datastores is Ian Varley's Master thesis, No Relation: The Mixed Blessings of Non-Relational Databases. Recommended. Also, read for how HBase stores data internally, and the section on - HBase Schema Design Case Studies. + .
@@ -41,7 +41,7 @@ <para>HBase schemas can be created or updated with <xref linkend="shell" /> or by using <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html">HBaseAdmin</link> in the Java API. </para> - <para>Tables must be disabled when making ColumnFamily modifications, for example.. + <para>Tables must be disabled when making ColumnFamily modifications, for example:</para> <programlisting> Configuration config = HBaseConfiguration.create(); HBaseAdmin admin = new HBaseAdmin(conf); @@ -56,7 +56,7 @@ admin.modifyColumn(table, cf2); // modifying existing ColumnFamily admin.enableTable(table); </programlisting> - </para>See <xref linkend="client_dependencies"/> for more information about configuring client connections. + <para>See <xref linkend="client_dependencies"/> for more information about configuring client connections.</para> <para>Note: online schema changes are supported in the 0.92.x codebase, but the 0.90.x codebase requires the table to be disabled. </para> @@ -98,7 +98,7 @@ admin.enableTable(table); Monotonically Increasing Row Keys/Timeseries Data - In the HBase chapter of Tom White's book Hadoop: The Definitive Guide (O'Reilly) there is a an optimization note on watching out for a phenomenon where an import process walks in lock-step with all clients in concert pounding one of the table's regions (and thus, a single node), then moving onto the next region, etc. With monotonically increasing row-keys (i.e., using a timestamp), this will happen. See this comic by IKai Lan on why monotonically increasing row keys are problematic in BigTable-like datastores: + In the HBase chapter of Tom White's book Hadoop: The Definitive Guide (O'Reilly) there is a an optimization note on watching out for a phenomenon where an import process walks in lock-step with all clients in concert pounding one of the table's regions (and thus, a single node), then moving onto the next region, etc. With monotonically increasing row-keys (i.e., using a timestamp), this will happen. See this comic by IKai Lan on why monotonically increasing row keys are problematic in BigTable-like datastores: monotonically increasing values are bad. The pile-up on a single region brought on by monotonically increasing keys can be mitigated by randomizing the input records to not be in sorted order, but in general it's best to avoid using a timestamp or a sequence (e.g. 1, 2, 3) as the row-key. @@ -107,7 +107,7 @@ admin.enableTable(table); successful example. It has a page describing the schema it uses in HBase. The key format in OpenTSDB is effectively [metric_type][event_timestamp], which would appear at first glance to contradict the previous advice about not using a timestamp as the key. However, the difference is that the timestamp is not in the lead position of the key, and the design assumption is that there are dozens or hundreds (or more) of different metric types. Thus, even with a continual stream of input data with a mix of metric types, the Puts are distributed across various points of regions in the table. - See HBase Schema Design Case Studies for some rowkey design examples. + See for some rowkey design examples.
@@ -119,7 +119,7 @@ admin.enableTable(table); are large, especially compared to the size of the cell value, then you may run up against some interesting scenarios. One such is the case described by Marc Limotte at the tail of - HBASE-3551 + HBASE-3551 (recommended!). Therein, the indices that are kept on HBase storefiles () to facilitate random access may end up occupyng large chunks of the HBase @@ -213,7 +213,7 @@ COLUMN CELL The most recent value for [key] in a table can be found by performing a Scan for [key] and obtaining the first record. Since HBase keys are in sorted order, this key sorts before any older row-keys for [key] and thus is first. - This technique would be used instead of using HBase Versioning where the intent is to hold onto all versions + This technique would be used instead of using where the intent is to hold onto all versions "forever" (or a very long time) and at the same time quickly obtain access to any other version by using the same Scan technique.
@@ -335,11 +335,11 @@ public static byte[][] getHexSplits(String startKey, String endKey, int numRegio Result, so anything that can be converted to an array of bytes can be stored as a value. Input could be strings, numbers, complex objects, or even images as long as they can rendered as bytes. - There are practical limits to the size of values (e.g., storing 10-50MB objects in HBase - would probably be too much to ask); search the mailing list for conversations on this topic. - All rows in HBase conform to the datamodel, and that includes - versioning. Take that into consideration when making your design, as well as block size for - the ColumnFamily. + There are practical limits to the size of values (e.g., storing 10-50MB objects in HBase would probably be too much to ask); + search the mailling list for conversations on this topic. All rows in HBase conform to the , and + that includes versioning. Take that into consideration when making your design, as well as block size for the ColumnFamily. + +
Counters @@ -389,10 +389,10 @@ public static byte[][] getHexSplits(String startKey, String endKey, int numRegio There is no single answer on the best way to handle this because it depends on... - Number of users - Data size and data arrival rate - Flexibility of reporting requirements (e.g., completely ad-hoc date selection vs. pre-configured ranges) - Desired execution speed of query (e.g., 90 seconds may be reasonable to some for an ad-hoc report, whereas it may be too long for others) + Number of users + Data size and data arrival rate + Flexibility of reporting requirements (e.g., completely ad-hoc date selection vs. pre-configured ranges) + Desired execution speed of query (e.g., 90 seconds may be reasonable to some for an ad-hoc report, whereas it may be too long for others) ... and solutions are also influenced by the size of the cluster and how much processing power you have to throw at the solution. Common techniques are in sub-sections below. This is a comprehensive, but not exhaustive, list of approaches. @@ -455,26 +455,26 @@ public static byte[][] getHexSplits(String startKey, String endKey, int numRegio can be approached. Note: this is just an illustration of potential approaches, not an exhaustive list. Know your data, and know your processing requirements. - It is highly recommended that you read the rest of the Schema Design Chapter first, before reading + It is highly recommended that you read the rest of the first, before reading these case studies. Thee following case studies are described: - Log Data / Timeseries Data - Log Data / Timeseries on Steroids - Customer/Order - Tall/Wide/Middle Schema Design - List Data + Log Data / Timeseries Data + Log Data / Timeseries on Steroids + Customer/Order + Tall/Wide/Middle Schema Design + List Data
Case Study - Log Data and Timeseries Data Assume that the following data elements are being collected. - Hostname - Timestamp - Log event - Value/message + Hostname + Timestamp + Log event + Value/message We can store them in an HBase table called LOG_DATA, but what will the rowkey be? From these attributes the rowkey will be some combination of hostname, timestamp, and log-event - but what specifically? @@ -531,9 +531,9 @@ long bucket = timestamp % numBuckets; Composite Rowkey With Hashes: - [MD5 hash of hostname] = 16 bytes - [MD5 hash of event-type] = 16 bytes - [timestamp] = 8 bytes + [MD5 hash of hostname] = 16 bytes + [MD5 hash of event-type] = 16 bytes + [timestamp] = 8 bytes Composite Rowkey With Numeric Substitution: @@ -541,17 +541,17 @@ long bucket = timestamp % numBuckets; For this approach another lookup table would be needed in addition to LOG_DATA, called LOG_TYPES. The rowkey of LOG_TYPES would be: - [type] (e.g., byte indicating hostname vs. event-type) - [bytes] variable length bytes for raw hostname or event-type. + [type] (e.g., byte indicating hostname vs. event-type) + [bytes] variable length bytes for raw hostname or event-type. A column for this rowkey could be a long with an assigned number, which could be obtained by using an HBase counter. So the resulting composite rowkey would be: - [substituted long for hostname] = 8 bytes - [substituted long for event type] = 8 bytes - [timestamp] = 8 bytes + [substituted long for hostname] = 8 bytes + [substituted long for event type] = 8 bytes + [timestamp] = 8 bytes In either the Hash or Numeric substitution approach, the raw values for hostname and event-type can be stored as columns. @@ -586,19 +586,19 @@ long bucket = timestamp % numBuckets; The Customer record type would include all the things that you’d typically expect: - Customer number - Customer name - Address (e.g., city, state, zip) - Phone numbers, etc. + Customer number + Customer name + Address (e.g., city, state, zip) + Phone numbers, etc. The Order record type would include things like: - Customer number - Order number - Sales date - A series of nested objects for shipping locations and line-items (see - for details) + Customer number + Order number + Sales date + A series of nested objects for shipping locations and line-items (see + for details) Assuming that the combination of customer number and sales order uniquely identify an order, these two attributes will compose @@ -614,14 +614,14 @@ reasonable spread in the keyspace, similar options appear: Composite Rowkey With Hashes: - [MD5 of customer number] = 16 bytes - [MD5 of order number] = 16 bytes + [MD5 of customer number] = 16 bytes + [MD5 of order number] = 16 bytes Composite Numeric/Hash Combo Rowkey: - [substituted long for customer number] = 8 bytes - [MD5 of order number] = 16 bytes + [substituted long for customer number] = 8 bytes + [MD5 of order number] = 16 bytes
@@ -631,15 +631,15 @@ reasonable spread in the keyspace, similar options appear: Customer Record Type Rowkey: - [customer-id] - [type] = type indicating ‘1’ for customer record type + [customer-id] + [type] = type indicating ‘1’ for customer record type Order Record Type Rowkey: - [customer-id] - [type] = type indicating ‘2’ for order record type - [order] + [customer-id] + [type] = type indicating ‘2’ for order record type + [order] The advantage of this particular CUSTOMER++ approach is that organizes many different record-types by customer-id @@ -665,23 +665,23 @@ reasonable spread in the keyspace, similar options appear: The SHIPPING_LOCATION's composite rowkey would be something like this: - [order-rowkey] - [shipping location number] (e.g., 1st location, 2nd, etc.) + [order-rowkey] + [shipping location number] (e.g., 1st location, 2nd, etc.) The LINE_ITEM table's composite rowkey would be something like this: - [order-rowkey] - [shipping location number] (e.g., 1st location, 2nd, etc.) - [line item number] (e.g., 1st lineitem, 2nd, etc.) + [order-rowkey] + [shipping location number] (e.g., 1st location, 2nd, etc.) + [line item number] (e.g., 1st lineitem, 2nd, etc.) Such a normalized model is likely to be the approach with an RDBMS, but that's not your only option with HBase. The cons of such an approach is that to retrieve information about any Order, you will need: - Get on the ORDER table for the Order - Scan on the SHIPPING_LOCATION table for that order to get the ShippingLocation instances - Scan on the LINE_ITEM for each ShippingLocation + Get on the ORDER table for the Order + Scan on the SHIPPING_LOCATION table for that order to get the ShippingLocation instances + Scan on the LINE_ITEM for each ShippingLocation ... granted, this is what an RDBMS would do under the covers anyway, but since there are no joins in HBase you're just more aware of this fact. @@ -693,23 +693,23 @@ reasonable spread in the keyspace, similar options appear: The Order rowkey was described above: - [order-rowkey] - [ORDER record type] + [order-rowkey] + [ORDER record type] The ShippingLocation composite rowkey would be something like this: - [order-rowkey] - [SHIPPING record type] - [shipping location number] (e.g., 1st location, 2nd, etc.) + [order-rowkey] + [SHIPPING record type] + [shipping location number] (e.g., 1st location, 2nd, etc.) The LineItem composite rowkey would be something like this: - [order-rowkey] - [LINE record type] - [shipping location number] (e.g., 1st location, 2nd, etc.) - [line item number] (e.g., 1st lineitem, 2nd, etc.) + [order-rowkey] + [LINE record type] + [shipping location number] (e.g., 1st location, 2nd, etc.) + [line item number] (e.g., 1st lineitem, 2nd, etc.)
@@ -720,21 +720,21 @@ reasonable spread in the keyspace, similar options appear: The LineItem composite rowkey would be something like this: - [order-rowkey] - [LINE record type] - [line item number] (e.g., 1st lineitem, 2nd, etc. - care must be taken that there are unique across the entire order) + [order-rowkey] + [LINE record type] + [line item number] (e.g., 1st lineitem, 2nd, etc. - care must be taken that there are unique across the entire order) ... and the LineItem columns would be something like this: - itemNumber - quantity - price - shipToLine1 (denormalized from ShippingLocation) - shipToLine2 (denormalized from ShippingLocation) - shipToCity (denormalized from ShippingLocation) - shipToState (denormalized from ShippingLocation) - shipToZip (denormalized from ShippingLocation) + itemNumber + quantity + price + shipToLine1 (denormalized from ShippingLocation) + shipToLine2 (denormalized from ShippingLocation) + shipToCity (denormalized from ShippingLocation) + shipToState (denormalized from ShippingLocation) + shipToZip (denormalized from ShippingLocation) The pros of this approach include a less complex object heirarchy, but one of the cons is that updating gets more @@ -789,7 +789,7 @@ reasonable spread in the keyspace, similar options appear: OpenTSDB is the best example of this case where a single row represents a defined time-range, and then discrete events are treated as columns. This approach is often more complex, and may require the additional complexity of re-writing your data, but has the advantage of being I/O efficient. For an overview of this approach, see - . + .
@@ -815,7 +815,7 @@ we could have something like: <FixedWidthUserName><FixedWidthValueId3>:"" (no value) -The other option we had was to do this entirely using: +The other option we had was to do this entirely using: <FixedWidthUserName><FixedWidthPageNum0>:<FixedWidthLength><FixedIdNextPageNum><ValueId1><ValueId2><ValueId3>... <FixedWidthUserName><FixedWidthPageNum1>:<FixedWidthLength><FixedIdNextPageNum><ValueId1><ValueId2><ValueId3>... @@ -827,8 +827,8 @@ So in one case reading the first thirty values would be: scan { STARTROW => 'FixedWidthUsername' LIMIT => 30} -And in the second case it would be - +And in the second case it would be + get 'FixedWidthUserName\x00\x00\x00\x00' diff --git a/src/main/docbkx/security.xml b/src/main/docbkx/security.xml index 47975228771..1bac920e455 100644 --- a/src/main/docbkx/security.xml +++ b/src/main/docbkx/security.xml @@ -517,7 +517,7 @@ You must configure HBase for secure or simple user access operation. Refer to the Secure Client Access to HBase or - Simple User Access to HBase + Simple User Access to HBase sections and complete all of the steps described there. @@ -779,14 +779,16 @@ Access control mechanisms are mature and fairly standardized in the relational d The HBase shell has been extended to provide simple commands for editing and updating user permissions. The following commands have been added for access control list management: - Grant - - + + + + Grant + grant <user|@group> <permissions> [ <table> [ <column family> [ <column qualifier> ] ] ] - - + + - <user|@group> is user or group (start with character '@'), Groups are created and manipulated via the Hadoop group mapping service. + <user|@group> is user or group (start with character '@'), Groups are created and manipulated via the Hadoop group mapping service. <permissions> is zero or more letters from the set "RWCA": READ('R'), WRITE('W'), CREATE('C'), ADMIN('A'). @@ -794,32 +796,32 @@ The HBase shell has been extended to provide simple commands for editing and upd Note: Grants and revocations of individual permissions on a resource are both accomplished using the grant command. A separate revoke command is also provided by the shell, but this is for fast revocation of all of a user's access rights to a given resource only. - - Revoke - - + + Revoke + revoke <user|@group> [ <table> [ <column family> [ <column qualifier> ] ] ] - + + + Alter + - Alter - - - The alter command has been extended to allow ownership assignment: + The alter command has been extended to allow ownership assignment: alter 'tablename', {OWNER => 'username|@group'} - + + + + User Permission + - User Permission - - - The user_permission command shows all access permissions for the current user for a given table: + The user_permission command shows all access permissions for the current user for a given table: user_permission <table> - +
@@ -829,12 +831,12 @@ The HBase shell has been extended to provide simple commands for editing and upd Bulk loading in secure mode is a bit more involved than normal setup, since the client has to transfer the ownership of the files generated from the mapreduce job to HBase. Secure bulk loading is implemented by a coprocessor, named SecureBulkLoadEndpoint. SecureBulkLoadEndpoint uses a staging directory "hbase.bulkload.staging.dir", which defaults to /tmp/hbase-staging/. The algorithm is as follows. - Create an hbase owned staging directory which is world traversable (-rwx--x--x, 711) /tmp/hbase-staging. - A user writes out data to his secure output directory: /user/foo/data - A call is made to hbase to create a secret staging directory - which is globally readable/writable (-rwxrwxrwx, 777): /tmp/hbase-staging/averylongandrandomdirectoryname - The user makes the data world readable and writable, then moves it - into the random staging directory, then calls bulkLoadHFiles() + Create an hbase owned staging directory which is world traversable (-rwx--x--x, 711) /tmp/hbase-staging. + A user writes out data to his secure output directory: /user/foo/data + A call is made to hbase to create a secret staging directory + which is globally readable/writable (-rwxrwxrwx, 777): /tmp/hbase-staging/averylongandrandomdirectoryname + The user makes the data world readable and writable, then moves it + into the random staging directory, then calls bulkLoadHFiles() @@ -870,9 +872,9 @@ The HBase shell has been extended to provide simple commands for editing and upd Visibility expressions like the above can be added when storing or mutating a cell using the API, - Mutation#setCellVisibility(new CellVisibility(String labelExpession)); - Where the labelExpression could be '( secret | topsecret ) & !probationary' - + Mutation#setCellVisibility(new CellVisibility(String labelExpession)); + Where the labelExpression could be '( secret | topsecret ) & !probationary' + We build the user's label set in the RPC context when a request is first received by the HBase RegionServer. How users are associated with labels is pluggable. The default plugin passes through labels specified in Authorizations added to the Get or Scan and checks those against the calling user's authenticated labels list. When client passes some labels for which the user is not authenticated, this default algorithm will drop those. One can pass a subset of user authenticated labels via the Scan/Get authorizations. @@ -896,7 +898,7 @@ The HBase shell has been extended to provide simple commands for editing and upd
-
+
User Label Association A set of labels can be associated with a user by using the API VisibilityClient#setAuths(Configuration conf, final String[] auths, final String user) diff --git a/src/main/docbkx/troubleshooting.xml b/src/main/docbkx/troubleshooting.xml index 7aebedaa99b..4c426d80ce9 100644 --- a/src/main/docbkx/troubleshooting.xml +++ b/src/main/docbkx/troubleshooting.xml @@ -235,7 +235,7 @@ export SERVER_GC_OPTS="$SERVER_GC_OPTS -XX:NewSize=64m -XX:MaxNewSize=64m" is generally used for questions on released versions of Apache HBase. Before going to the mailing list, make sure your question has not already been answered by searching the mailing list archives first. Use . - Take some time crafting your questionSee Getting Answers; a quality question that includes all context and + Take some time crafting your questionSee Getting Answers; a quality question that includes all context and exhibits evidence the author has tried to find answers in the manual and out on lists is more likely to get a prompt response. @@ -360,15 +360,15 @@ hadoop@sv4borg12:~$ jps In order, we see a: - Hadoop TaskTracker, manages the local Childs - HBase RegionServer, serves regions - Child, its MapReduce task, cannot tell which type exactly - Hadoop TaskTracker, manages the local Childs - Hadoop DataNode, serves blocks - HQuorumPeer, a ZooKeeper ensemble member - Jps, well… it’s the current process - ThriftServer, it’s a special one will be running only if thrift was started - jmx, this is a local process that’s part of our monitoring platform ( poorly named maybe). You probably don’t have that. + Hadoop TaskTracker, manages the local Childs + HBase RegionServer, serves regions + Child, its MapReduce task, cannot tell which type exactly + Hadoop TaskTracker, manages the local Childs + Hadoop DataNode, serves blocks + HQuorumPeer, a ZooKeeper ensemble member + Jps, well… it’s the current process + ThriftServer, it’s a special one will be running only if thrift was started + jmx, this is a local process that’s part of our monitoring platform ( poorly named maybe). You probably don’t have that. @@ -620,21 +620,19 @@ Harsh J investigated the issue as part of the mailing list thread
Client running out of memory though heap size seems to be stable (but the off-heap/direct heap keeps growing) - You are likely running into the issue that is described and worked through in the - mail thread HBase, mail # user - Suspected memory leak and continued over in HBase, mail # dev - FeedbackRe: Suspected memory leak. A workaround is passing - your client-side JVM a reasonable value for -XX:MaxDirectMemorySize. By - default, the MaxDirectMemorySize is equal to your -Xmx max - heapsize setting (if -Xmx is set). Try seting it to something smaller (for - example, one user had success setting it to 1g when they had a client-side heap - of 12g). If you set it too small, it will bring on FullGCs so keep - it a bit hefty. You want to make this setting client-side only especially if you are running - the new experimental server-side off-heap cache since this feature depends on being able to - use big direct buffers (You may have to keep separate client-side and server-side config - dirs). + +You are likely running into the issue that is described and worked through in +the mail thread HBase, mail # user - Suspected memory leak +and continued over in HBase, mail # dev - FeedbackRe: Suspected memory leak. +A workaround is passing your client-side JVM a reasonable value for -XX:MaxDirectMemorySize. By default, +the MaxDirectMemorySize is equal to your -Xmx max heapsize setting (if -Xmx is set). +Try seting it to something smaller (for example, one user had success setting it to 1g when +they had a client-side heap of 12g). If you set it too small, it will bring on FullGCs so keep +it a bit hefty. You want to make this setting client-side only especially if you are running the new experiemental +server-side off-heap cache since this feature depends on being able to use big direct buffers (You may have to keep +separate client-side and server-side config dirs). + +
Client Slowdown When Calling Admin Methods (flush, compact, etc.) @@ -753,7 +751,7 @@ Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. /<HLog> (WAL HLog files for the RegionServer) - See the HDFS User Guide for other non-shell diagnostic + See the HDFS User Guide for other non-shell diagnostic utilities like fsck.
@@ -926,25 +924,26 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi Since the RegionServer's local ZooKeeper client cannot send heartbeats, the session times out. By design, we shut down any node that isn't able to contact the ZooKeeper ensemble after getting a timeout so that it stops serving data that may already be assigned elsewhere. - + - Make sure you give plenty of RAM (in hbase-env.sh), the default of 1GB won't be able to sustain long running imports. - Make sure you don't swap, the JVM never behaves well under swapping. - Make sure you are not CPU starving the RegionServer thread. For example, if you are running a MapReduce job using 6 CPU-intensive tasks on a machine with 4 cores, you are probably starving the RegionServer enough to create longer garbage collection pauses. - Increase the ZooKeeper session timeout + Make sure you give plenty of RAM (in hbase-env.sh), the default of 1GB won't be able to sustain long running imports. + Make sure you don't swap, the JVM never behaves well under swapping. + Make sure you are not CPU starving the RegionServer thread. For example, if you are running a MapReduce job using 6 CPU-intensive tasks on a machine with 4 cores, you are probably starving the RegionServer enough to create longer garbage collection pauses. + Increase the ZooKeeper session timeout - If you wish to increase the session timeout, add the following to your hbase-site.xml to increase the timeout from the default of 60 seconds to 120 seconds. - -<property> - <name>zookeeper.session.timeout</name> - <value>1200000</value> -</property> -<property> - <name>hbase.zookeeper.property.tickTime</name> - <value>6000</value> -</property> + If you wish to increase the session timeout, add the following to your hbase-site.xml to increase the timeout from the default of 60 seconds to 120 seconds. + + + + zookeeper.session.timeout + 1200000 + + + hbase.zookeeper.property.tickTime + 6000 +]]> - + Be aware that setting a higher timeout means that the regions served by a failed RegionServer will take at least that amount of time to be transfered to another RegionServer. For a production system serving live requests, we would instead @@ -954,8 +953,8 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi If this is happening during an upload which only happens once (like initially loading all your data into HBase), consider bulk loading. - See for other general information about ZooKeeper troubleshooting. -
+See for other general information about ZooKeeper troubleshooting. +
NotServingRegionException This exception is "normal" when found in the RegionServer logs at DEBUG level. This exception is returned back to the client @@ -970,7 +969,7 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi RegionServer is not using the name given it by the master; double entry in master listing of servers for gorey details.
-
+
Logs flooded with '2011-01-10 12:40:48,407 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor' messages We are not using the native versions of compression @@ -992,7 +991,7 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi
Shutdown Errors - +
@@ -1020,7 +1019,7 @@ ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expi
Shutdown Errors - +
diff --git a/src/main/docbkx/upgrading.xml b/src/main/docbkx/upgrading.xml index 7711614d21b..758d6d4317c 100644 --- a/src/main/docbkx/upgrading.xml +++ b/src/main/docbkx/upgrading.xml @@ -225,8 +225,8 @@ Now start up hbase-0.96.0.
-
Troubleshooting -
Old Client connecting to 0.96 cluster +
Troubleshooting +
Old Client connecting to 0.96 cluster It will fail with an exception like the below. Upgrade. 17:22:15 Exception in thread "main" java.lang.IllegalArgumentException: Not a host:port pair: PBUF 17:22:15 * @@ -266,20 +266,20 @@ If you've not patience, here are the important things to know upgrading. -Once you upgrade, you can’t go back. +Once you upgrade, you can’t go back. - -MSLAB is on by default. Watch that heap usage if you have a lot of regions. + +MSLAB is on by default. Watch that heap usage if you have a lot of regions. - + Distributed splitting is on by defaul. It should make region server failover faster. - - + + There’s a separate tarball for security. - - + + If -XX:MaxDirectMemorySize is set in your hbase-env.sh, it’s going to enable the experimental off-heap cache (You may not want this). - + @@ -301,7 +301,7 @@ This means you cannot go back to 0.90.x once you’ve started HBase 0.92.0 over In 0.92.0, the hbase.hregion.memstore.mslab.enabled flag is set to true (See ). In 0.90.x it was false. When it is enabled, memstores will step allocate memory in MSLAB 2MB chunks even if the memstore has zero or just a few small elements. This is fine usually but if you had lots of regions per regionserver in a 0.90.x cluster (and MSLAB was off), -you may find yourself OOME'ing on upgrade because the thousands of regions * number of column families * 2MB MSLAB (at a minimum) +you may find yourself OOME'ing on upgrade because the thousands of regions * number of column families * 2MB MSLAB (at a minimum) puts your heap over the top. Set hbase.hregion.memstore.mslab.enabled to false or set the MSLAB size down from 2MB by setting hbase.hregion.memstore.mslab.chunksize to something less. diff --git a/src/main/docbkx/zookeeper.xml b/src/main/docbkx/zookeeper.xml index 4a257b59452..d4d87749784 100644 --- a/src/main/docbkx/zookeeper.xml +++ b/src/main/docbkx/zookeeper.xml @@ -219,7 +219,7 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper standalone Zookeeper quorum) for ease of learning. -
Operating System Prerequisites
+
Operating System Prerequisites You need to have a working Kerberos KDC setup. For @@ -283,7 +283,7 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper We'll refer to this JAAS configuration file as $CLIENT_CONF below. - +
HBase-managed Zookeeper Configuration @@ -312,10 +312,10 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper }; - where the $PATH_TO_HBASE_KEYTAB and + where the $PATH_TO_HBASE_KEYTAB and $PATH_TO_ZOOKEEPER_KEYTAB files are what you created above, and $HOST is the hostname for that - node. + node. The Server section will be used by the Zookeeper quorum server, while the @@ -342,9 +342,9 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" - where $HBASE_SERVER_CONF and + where $HBASE_SERVER_CONF and $CLIENT_CONF are the full paths to the - JAAS configuration files created above. + JAAS configuration files created above. Modify your hbase-site.xml on each node that will run zookeeper, master or regionserver to contain: @@ -537,10 +537,10 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper
Configuration from Scratch - This has been tested on the current standard Amazon + This has been tested on the current standard Amazon Linux AMI. First setup KDC and principals as described above. Next checkout code and run a sanity - check. + check. git clone git://git.apache.org/hbase.git @@ -548,9 +548,9 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper mvn clean test -Dtest=TestZooKeeperACL - Then configure HBase as described above. - Manually edit target/cached_classpath.txt (see below).. - + Then configure HBase as described above. + Manually edit target/cached_classpath.txt (see below): + bin/hbase zookeeper & bin/hbase master & @@ -582,14 +582,16 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper programmatically - This would avoid the need for a separate Hadoop jar + This would avoid the need for a separate Hadoop jar that fixes HADOOP-7070. +
Elimination of <code>kerberos.removeHostFromPrincipal</code> and <code>kerberos.removeRealmFromPrincipal</code> +