From 537463ec95af12e887702d535c8b0523ce183a54 Mon Sep 17 00:00:00 2001 From: Michael Stack Date: Fri, 22 Apr 2011 19:07:56 +0000 Subject: [PATCH] Added a head-up to preface that user is about to enter the realm of distributed computing, added how to enable rpc logging, added note to decommissioning server that balacner should be off, and converted links to xrefs when they were linkends git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1095988 13f79535-47bb-0310-9956-ffa450edef68 --- src/docbkx/book.xml | 66 ++++++++++++++++++++++------------ src/docbkx/configuration.xml | 18 +++++----- src/docbkx/getting_started.xml | 46 +++++++++--------------- src/docbkx/performance.xml | 44 +++++++++++++++++------ src/docbkx/preface.xml | 21 +++++++++++ src/docbkx/shell.xml | 2 +- src/docbkx/upgrading.xml | 5 ++- 7 files changed, 126 insertions(+), 76 deletions(-) diff --git a/src/docbkx/book.xml b/src/docbkx/book.xml index bb2017afe71..3d4e3ce56a2 100644 --- a/src/docbkx/book.xml +++ b/src/docbkx/book.xml @@ -148,7 +148,7 @@ throws InterruptedException, IOException { Schema Creation - HBase schemas can be created or updated through the HBase shell + HBase schemas can be created or updated with or by using HBaseAdmin in the Java API. @@ -197,7 +197,7 @@ throws InterruptedException, IOException { the case described by Marc Limotte at the tail of HBASE-3551 (recommended!). - Therein, the indices that are kept on HBase storefiles (HFiles) + Therein, the indices that are kept on HBase storefiles () to facilitate random access may end up occupyng large chunks of the HBase allotted RAM because the cell value coordinates are large. Mark in the above cited comment suggests upping the block size so @@ -213,7 +213,7 @@ throws InterruptedException, IOException { The number of row versions to store is configured per column family via HColumnDescriptor. The default is 3. - This is an important parameter because as described in the Data Model + This is an important parameter because as described in section HBase does not overwrite row values, but rather stores different values per row by time (and qualifier). Excess versions are removed during major compactions. The number of versions may need to be increased or decreased depending on application needs. @@ -248,7 +248,7 @@ throws InterruptedException, IOException { Size of the compaction queue. This is the number of stores in the region that have been targeted for compaction.
<varname>hbase.regionserver.fsReadLatency_avg_time</varname> - Filesystem read latency (ms) + Filesystem read latency (ms). This is the average time to read from HDFS.
<varname>hbase.regionserver.fsReadLatency_num_ops</varname> TODO @@ -294,11 +294,10 @@ throws InterruptedException, IOException { Data Model - In short, applications store data into HBase tables. - Tables are made of rows and columns. - All columns in HBase belong to a particular - column family. - Table cells -- the intersection of row and column + In short, applications store data into an HBase table. + Tables are made of rows and columns. + All columns in HBase belong to a particular column family. + Table cells -- the intersection of row and column coordinates -- are versioned. A cell’s content is an uninterpreted array of bytes. @@ -709,7 +708,7 @@ throws InterruptedException, IOException { Administrative functions are handled through HBaseAdmin
Connections - For connection configuration information, see the configuration section. + For connection configuration information, see . HTable instances are not thread-safe. When creating HTable instances, it is advisable to use the same HBaseConfiguration @@ -728,7 +727,8 @@ HTable table2 = new HTable(conf2, "myTable");
WriteBuffer and Batch Methods - If autoflush is turned off on HTable, + If is turned off on + HTable, Puts are sent to region servers when the writebuffer is filled. The writebuffer is 2MB by default. Before an HTable instance is discarded, either close() or @@ -813,7 +813,7 @@ HTable table2 = new HTable(conf2, "myTable"); participate. The RegionServer splits a region, offlines the split region and then adds the daughter regions to META, opens daughters on the parent's hosting RegionServer and then reports the split to the - Master. See Managed Splitting for how to manually manage + Master. See for how to manually manage splits (and for why you might do this)
@@ -872,7 +872,7 @@ HTable table2 = new HTable(conf2, "myTable"); For a description of how a minor compaction picks files to compact, see the ascii diagram in the Store source code. After a major compaction runs there will be a single storefile per store, and this will help performance usually. Caution: major compactions rewrite all of the stores data and on a loaded system, this may not be tenable; - major compactions will usually have to be managed on large systems. + major compactions will usually have to be on large systems.
@@ -888,7 +888,7 @@ HTable table2 = new HTable(conf2, "myTable"); Purpose Each RegionServer adds updates (Puts, Deletes) to its write-ahead log (WAL) - first, and then to the MemStore for the affected Store. + first, and then to the for the affected . This ensures that HBase has durable writes. Without WAL, there is the possibility of data loss in the case of a RegionServer failure before each MemStore is flushed and new StoreFiles are written. HLog is the HBase WAL implementation, and there is one HLog instance per RegionServer. @@ -1090,7 +1090,7 @@ HTable table2 = new HTable(conf2, "myTable");
HFile Tool - See HFile Tool. + See .
WAL Tools @@ -1113,10 +1113,31 @@ HTable table2 = new HTable(conf2, "myTable");
Compression Tool - See Compression Tool. + See .
Node Decommission - Since HBase 0.90.2, you can have a node gradually shed its load and then shutdown using the + You can stop an individual regionserver by running the following + script in the HBase directory on the particular node: + $ ./bin/hbase-daemon.sh stop regionserver + The regionserver will first close all regions and then shut itself down. + On shutdown, the regionserver's ephemeral node in ZooKeeper will expire. + The master will notice the regionserver gone and will treat it as + a 'crashed' server; it will reassign the nodes the regionserver was carrying. + Disable the Load Balancer before Decommissioning a node + If the load balancer runs while a node is shutting down, then + there could be contention between the Load Balancer and the + Master's recovery of the just decommissioned regionserver. + Avoid any problems by disabling the balancer first. + See below. + + + + + A downside to the above stop of a regionserver is that regions could be offline for + a good period of time. Regions are closed in order. If many regions on the server, the + first region to close may not be back online until all regions close and after the master + notices the regionserver's znode gone. In HBase 0.90.2, we added facility for having + a node gradually shed its load and then shutdown itself down. HBase 0.90.2 added the graceful_stop.sh script. Here is its usage: $ ./bin/graceful_stop.sh Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thrift] [--rest] &hostname> @@ -1152,7 +1173,7 @@ Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thri RegionServer gone but all regions will have already been redeployed and because the RegionServer went down cleanly, there will be no WAL logs to split. - Load Balancer + Load Balancer It is assumed that the Region Load Balancer is disabled while the graceful_stop script runs (otherwise the balancer @@ -1270,7 +1291,7 @@ false LZO - See LZO Compression above. + See above.
@@ -1285,7 +1306,7 @@ false available on the CLASSPATH; in this case it will use native compressors instead (If the native libs are NOT present, you will see lots of Got brand-new compressor - reports in your logs; see FAQ). + reports in your logs; see ). @@ -1309,7 +1330,7 @@ false Not really. SQL-ish support for HBase via Hive is in development, however Hive is based on MapReduce which is not generally suitable for low-latency requests. - See the Data Model section for examples on the HBase client. + See the section for examples on the HBase client. @@ -1320,7 +1341,7 @@ false HDFS is a distributed file system that is well suited for the storage of large files. It's documentation states that it is not, however, a general purpose file system, and does not provide fast individual record lookups in files. HBase, on the other hand, is built on top of HDFS and provides fast record lookups (and updates) for large tables. This can sometimes be a point of conceptual confusion. - See the Data Model and Architecture sections for more information on how HBase achieves its goals. + See the and sections for more information on how HBase achieves its goals. @@ -1406,6 +1427,7 @@ When I build, why do I always get Unable to find resource 'VM_global_libra <link xlink:href="https://github.com/brianfrankcooper/YCSB/">YCSB: The Yahoo! Cloud Serving Benchmark</link> and HBase + TODO: Describe how YCSB is poor for putting up a decent cluster load. TODO: Describe setup of YCSB for HBase Ted Dunning redid YCSB so its mavenized and added facility for verifying workloads. See Ted Dunning's YCSB. diff --git a/src/docbkx/configuration.xml b/src/docbkx/configuration.xml index bdcc0bcf4aa..245266798d2 100644 --- a/src/docbkx/configuration.xml +++ b/src/docbkx/configuration.xml @@ -40,7 +40,7 @@ to ensure well-formedness of your document after an edit session. for HBase, site specific customizations go into the file conf/hbase-site.xml. For the list of configurable properties, see - Default HBase Configurations + below or view the raw hbase-default.xml source file in the HBase source code at src/main/resources. @@ -99,10 +99,10 @@ to ensure well-formedness of your document after an edit session.
Required Configurations - See the Requirements section. + See . It lists at least two required configurations needed running HBase bearing - load: i.e. file descriptors ulimit and - dfs.datanode.max.xcievers. + load: i.e. and + .
@@ -185,10 +185,10 @@ to ensure well-formedness of your document after an edit session. fixup on the new machine. In versions since HBase 0.90.0, we should fail in a way that makes it plain what the problem is, but maybe not. Remember you read this paragraphSee - hbase.regionserver.codecs + for a feature to help protect against failed LZO install. - See also the Compression Appendix + See also at the tail of this book.
@@ -303,11 +303,11 @@ of all regions. (Invocation will also factor in any hbase-default.xml found; an hbase-default.xml ships inside the hbase.X.X.X.jar). It is also possible to specify configuration directly without having to read from a - hbase-site.xml. For example, to set the - zookeeper ensemble for the cluster programmatically do as follows: + hbase-site.xml. For example, to set the ZooKeeper + ensemble for the cluster programmatically do as follows: Configuration config = HBaseConfiguration.create(); config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zookeeper locally - If multiple zookeeper instances make up your zookeeper ensemble, + If multiple ZooKeeper instances make up your zookeeper ensemble, they may be specified in a comma-separated list (just as in the hbase-site.xml file). This populated Configuration instance can then be passed to an HTable, diff --git a/src/docbkx/getting_started.xml b/src/docbkx/getting_started.xml index 3d5debbaaaf..9fc6ed47853 100644 --- a/src/docbkx/getting_started.xml +++ b/src/docbkx/getting_started.xml @@ -12,9 +12,9 @@
Introduction - Quick Start will get you up and + will get you up and running on a single-node instance of HBase using the local filesystem. The - Not-so-quick Start Guide describes setup + describes setup of HBase in distributed mode running on top of HDFS.
@@ -23,7 +23,7 @@ This guide describes setup of a standalone HBase instance that uses the local filesystem. It leads you through creating a table, inserting - rows via the HBase Shell, and then cleaning + rows via the HBase shell, and then cleaning up and shutting down your standalone HBase instance. The below exercise should take no more than ten minutes (not including download time). @@ -97,8 +97,7 @@ starting Master, logging to logs/hbase-user-master-example.org.out Shell Exercises - Connect to your running HBase via the HBase - Shell. + Connect to your running HBase via the shell. $ ./bin/hbase shell HBase Shell; enter 'help<RETURN>' for list of supported commands. @@ -114,8 +113,7 @@ hbase(main):001:0> HBase shell; in particular note how table names, rows, and columns, etc., must be quoted. - Create a table named test with a single column family named cf. + Create a table named test with a single column family named cf. Verify its creation by listing all tables and then insert some values. @@ -133,8 +131,7 @@ hbase(main):006:0> put 'test', 'row3', 'cf:c', 'value3' Above we inserted 3 values, one at a time. The first insert is at row1, column cf:a with a value of - value1. Columns in HBase are comprised of a column family prefix -- + value1. Columns in HBase are comprised of a column family prefix -- cf in this example -- followed by a colon and then a column qualifier suffix (a in this case). @@ -182,8 +179,7 @@ stopping hbase............... Where to go next The above described standalone setup is good for testing and - experiments only. Move on to the next section, the Not-so-quick Start Guide where we'll go into + experiments only. Next move on to where we'll go into depth on the different HBase run modes, requirements and critical configurations needed setting up a distributed HBase deploy.
@@ -437,9 +433,7 @@ stopping hbase...............
HBase run modes: Standalone and Distributed - HBase has two run modes: standalone and distributed. Out of the box, HBase runs in + HBase has two run modes: and . Out of the box, HBase runs in standalone mode. To set up a distributed deploy, you will need to configure HBase by editing files in the HBase conf directory. @@ -456,7 +450,7 @@ stopping hbase............... Standalone HBase This is the default mode. Standalone mode is what is described - in the quickstart section. In + in the section. In standalone mode, HBase does not use HDFS -- it uses the local filesystem instead -- and it runs all HBase daemons and a local zookeeper all up in the same JVM. Zookeeper binds to a well known port @@ -485,8 +479,7 @@ stopping hbase............... verification and exploration of your install, whether a pseudo-distributed or fully-distributed configuration is described in a - section that follows, Running and Confirming - your Installation. The same verification script applies to both + section that follows, . The same verification script applies to both deploy types.
@@ -499,10 +492,8 @@ stopping hbase............... Once you have confirmed your HDFS setup, edit conf/hbase-site.xml. This is the file into - which you add local customizations and overrides for Default HBase - Configurations and HDFS - Client Configurations. Point HBase at the running Hadoop HDFS + which you add local customizations and overrides for + and . Point HBase at the running Hadoop HDFS instance by setting the hbase.rootdir property. This property points HBase at the Hadoop filesystem instance to use. For example, adding the properties below to your @@ -543,8 +534,7 @@ stopping hbase............... want to connect from a remote location. - Now skip to Running and Confirming - your Installation for how to start and verify your + Now skip to for how to start and verify your pseudo-distributed install. See Pseudo-distributed @@ -594,8 +584,7 @@ stopping hbase............... In addition, a fully-distributed mode requires that you modify conf/regionservers. The - regionservers file + file lists all hosts that you would have running HRegionServers, one host per line (This file in HBase is like the Hadoop slaves @@ -634,9 +623,7 @@ stopping hbase............... by setting the hbase.zookeeper.property.clientPort property. For all default values used by HBase, including ZooKeeper - configuration, see the section Default HBase - Configurations. Look for the + configuration, see . Look for the hbase.zookeeper.property prefix For the full list of ZooKeeper configurations, see ZooKeeper's zoo.cfg. HBase does not ship @@ -835,8 +822,7 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper - Once HBase has started, see the Shell Exercises section for how to + Once HBase has started, see the for how to create tables, add data, scan your insertions, and finally disable and drop your tables. diff --git a/src/docbkx/performance.xml b/src/docbkx/performance.xml index 34df654f64f..259949eef42 100644 --- a/src/docbkx/performance.xml +++ b/src/docbkx/performance.xml @@ -15,6 +15,20 @@ factors involved; RAM, compression, JVM settings, etc. Afterward, come back here for more pointers. + Enabling RPC-level logging + Enabling the RPC-level logging on a regionserver can often given + insight on timings at the server. Once enabled, the amount of log + spewed is voluminous. It is not recommended that you leave this + logging on for more than short bursts of time. To enable RPC-level + logging, browse to the regionserver UI and click on + Log Level. Set the log level to DEBUG for the package + org.apache.hadoop.ipc (Thats right, for + hadoop.ipc, NOT, hbase.ipc). Then tail the regionservers log. + Analyze. + To disable, set the logging level back to WARN level. + + +
Java @@ -46,16 +60,14 @@
Configurations - See the section on recommended - configurations. + See .
Number of Regions - The number of regions for an HBase table is driven by the filesize. Also, see the architecture - section on region size + The number of regions for an HBase table is driven by the . Also, see the architecture + section on
@@ -68,18 +80,28 @@
Compression - - Production systems should use compression such as LZO compression with their column family + Production systems should use compression such as compression with their column family definitions.
+ +
+ <varname>hbase.regionserver.handler.count</varname> + This setting is in essence sets how many requests are + concurrently being processed inside the regionserver at any + one time. If set too high, then throughput may suffer as + the concurrent requests contend; if set too low, requests will + be stuck waiting to get into the machine. You can get a + sense of whether you have too little or too many handlers by + + on an individual regionserver then tailing its logs. +
+
Number of Column Families - See the section on Number of Column - Families. + See .
diff --git a/src/docbkx/preface.xml b/src/docbkx/preface.xml index 92b3160be4b..5ff7e991f5a 100644 --- a/src/docbkx/preface.xml +++ b/src/docbkx/preface.xml @@ -23,4 +23,25 @@ hope to fill in the holes with time. Feel free to add to this book by adding a patch to an issue up in the HBase JIRA. + + + Heads-up + + If this is your first foray into the wonderful world of + Distributed Computing, then you are in for + some interesting times. First off, distributed systems are + hard; making a distributed system hum requires a disparate + skillset that needs span systems (hardware and software) and + networking. Your cluster' operation can hiccup because of any + of a myriad set of reasons from bugs in HBase itself through misconfigurations + -- misconfiguration of HBase but also operating system misconfigurations -- + through to hardware problems whether it be a bug in your network card + drivers or an underprovisioned RAM bus (to mention two recent + examples of hardware issues that manifested as "HBase is slow"). + You will also need to do a recalibration if up to this your + computing has been bound to a single box. Here is one good + starting point: + Fallacies of Distributed Computing. + + diff --git a/src/docbkx/shell.xml b/src/docbkx/shell.xml index f7f3c6f62d1..14e9da864a6 100644 --- a/src/docbkx/shell.xml +++ b/src/docbkx/shell.xml @@ -24,7 +24,7 @@ arguments are entered into the HBase shell; in particular note how table names, rows, and columns, etc., must be quoted. - See Shell Exercises + See for example basic shell operation.
Scripting diff --git a/src/docbkx/upgrading.xml b/src/docbkx/upgrading.xml index 2c8ca6b7d22..a2aac92bc5f 100644 --- a/src/docbkx/upgrading.xml +++ b/src/docbkx/upgrading.xml @@ -9,8 +9,7 @@ xmlns:db="http://docbook.org/ns/docbook"> Upgrading - Review the requirements - section above, in particular the section on Hadoop version. + Review , in particular the section on Hadoop version.
Upgrading to HBase 0.90.x from 0.20.x or 0.89.x @@ -30,7 +29,7 @@ HBase jar and read from there. If you would like to review the content of this file, see it in the src tree at src/main/resources/hbase-default.xml or - see Default HBase Configurations. + see . Finally, if upgrading from 0.20.x, check your