diff --git a/src/main/docbkx/configuration.xml b/src/main/docbkx/configuration.xml index 8ffdd8a6645..8c7406a44a7 100644 --- a/src/main/docbkx/configuration.xml +++ b/src/main/docbkx/configuration.xml @@ -65,8 +65,10 @@ to ensure well-formedness of your document after an edit session.
Java - Just like Hadoop, HBase requires at least java 6 from - Oracle. + Just like Hadoop, HBase requires at least Java 6 from + Oracle. + Java 7 should work and can even be faster than Java 6, but almost all testing + has been done on the latter at this point.
@@ -701,6 +703,8 @@ stopping hbase............... Shutdown can take a moment to
Client configuration and dependencies connecting to an HBase cluster + If you are running HBase in standalone mode, you don't need to configure anything for your client to work + provided that they are all on the same machine. Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for current critical locations. ZooKeeper is where all these values are kept. Thus clients diff --git a/src/main/docbkx/ops_mgt.xml b/src/main/docbkx/ops_mgt.xml index 272083fc806..84e943a00ee 100644 --- a/src/main/docbkx/ops_mgt.xml +++ b/src/main/docbkx/ops_mgt.xml @@ -490,6 +490,28 @@ false
+
+ Adding a New Node + Adding a new regionserver in HBase is essentially free, you simply start it like this: + $ ./bin/hbase-daemon.sh start regionserver + and it will register itself with the master. Ideally you also started a DataNode on the same + machine so that the RS can eventually start to have local files. If you rely on ssh to start your + daemons, don't forget to add the new hostname in conf/regionservers on the master. + + At this point the region server isn't serving data because no regions have moved to it yet. If the balancer is + enabled, it will start moving regions to the new RS. On a small/medium cluster this can have a very adverse effect + on latency as a lot of regions will be offline at the same time. It is thus recommended to disable the balancer + the same way it's done when decommissioning a node and move the regions manually (or even better, using a script + that moves them one by one). + + The moved regions will all have 0% locality and won't have any blocks in cache so the region server will have + to use the network to serve requests. Apart from resulting in higher latency, it may also be able to use all of + your network card's capacity. For practical purposes, consider that a standard 1GigE NIC won't be able to read + much more than 100MB/s. In this case, or if you are in a OLAP environment and require having + locality, then it is recommended to major compact the moved regions. + + +
diff --git a/src/main/docbkx/performance.xml b/src/main/docbkx/performance.xml index 29d69612b58..9b5106f27e5 100644 --- a/src/main/docbkx/performance.xml +++ b/src/main/docbkx/performance.xml @@ -684,7 +684,7 @@ the data will still be read. A fairly common question on the dist-list is why HBase isn't as performant as HDFS files in a batch context (e.g., as a MapReduce source or sink). The short answer is that HBase is doing a lot more than HDFS (e.g., reading the KeyValues, returning the most current row or specified timestamps, etc.), and as such HBase is 4-5 times slower than HDFS in this - processing context. Not that there isn't room for improvement (and this gap will, over time, be reduced), but HDFS + processing context. There is room for improvement and this gap will, over time, be reduced, but HDFS will always be faster in this use-case.
@@ -700,6 +700,24 @@ the data will still be read. +
Collocating HBase and MapReduce + It is often recommended to have different clusters for HBase and MapReduce. A better qualification of this is: + don't collocate a HBase that serves live requests with a heavy MR workload. OLTP and OLAP-optimized systems have + conflicting requirements and one will lose to the other, usually the former. For example, short latency-sensitive + disk reads will have to wait in line behind longer reads that are trying to squeeze out as much throughput as + possible. MR jobs that write to HBase will also generate flushes and compactions, which will in turn invalidate + blocks in the . + + If you need to process the data from your live HBase cluster in MR, you can ship the deltas with + or use replication to get the new data in real time on the OLAP cluster. In the worst case, if you really need to + collocate both, set MR to use less Map and Reduce slots than you'd normally configure, possibly just one. + + When HBase is used for OLAP operations, it's preferable to set it up in a hardened way like configuring the ZooKeeper session + timeout higher and giving more memory to the MemStores (the argument being that the Block Cache won't be used much + since the workloads are usually long scans). + +
+
Case Studies For Performance and Troubleshooting Case Studies, see . diff --git a/src/main/docbkx/schema_design.xml b/src/main/docbkx/schema_design.xml index c6f9b6d2636..765a8f7ddb7 100644 --- a/src/main/docbkx/schema_design.xml +++ b/src/main/docbkx/schema_design.xml @@ -180,6 +180,21 @@ byte[] sbDigest = Bytes.toBytes(sDigest); System.out.println("md5 digest as string length: " + sbDigest.length); // returns 26 + Unfortunately, using a binary representation of a type will make your data harder to read outside of your code. For example, + this is what you will see in the shell when you increment a value: + +hbase(main):001:0> incr 't', 'r', 'f:q', 1 +COUNTER VALUE = 1 + +hbase(main):002:0> get 't', 'r' +COLUMN CELL + f:q timestamp=1369163040570, value=\x00\x00\x00\x00\x00\x00\x00\x01 +1 row(s) in 0.0310 seconds + + The shell makes a best effort to print a string, and it this case it decided to just print the hex. The same will + happen to your row keys inside the region names. It can be okay if you know what's being stored, but it might also + be unreadable if arbitrary data can be put in the same cells. This is the main trade-off. +
diff --git a/src/main/docbkx/troubleshooting.xml b/src/main/docbkx/troubleshooting.xml index e2a8078284f..f19fc75c152 100644 --- a/src/main/docbkx/troubleshooting.xml +++ b/src/main/docbkx/troubleshooting.xml @@ -740,6 +740,10 @@ Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. See the HDFS User Guide for other non-shell diagnostic utilities like fsck. +
+ Zero size HLogs with data in them + Problem: when getting a listing of all the files in a region server's .logs directory, one file has a size of 0 but it contains data. + Answer: It's an HDFS quirk. A file that's currently being to will appear to have a size of 0 but once it's closed it will show its true size
Use Cases Two common use-cases for querying HDFS for HBase objects is research the degree of uncompaction of a table. If there are a large number of StoreFiles for each ColumnFamily it could