From bfa4e138c7b770094664d62ff52c0ebad59cec67 Mon Sep 17 00:00:00 2001 From: Michael Stack Date: Tue, 6 Dec 2011 23:13:37 +0000 Subject: [PATCH] HBASE-4376 Document mutual authentication between HBase and Zookeeper using SASL git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1211224 13f79535-47bb-0310-9956-ffa450edef68 --- src/docbkx/configuration.xml | 406 +++++++++++++++++++++++++++++++++++ 1 file changed, 406 insertions(+) diff --git a/src/docbkx/configuration.xml b/src/docbkx/configuration.xml index 08dab691ba5..4b9cdb5edf4 100644 --- a/src/docbkx/configuration.xml +++ b/src/docbkx/configuration.xml @@ -722,6 +722,412 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper for more information on ZooKeeper sizing. + + +
+ SASL Authentication with ZooKeeper + Newer releases of HBase (>= 0.92) will + support connecting to a ZooKeeper Quorum that supports + SASL authentication (which is available in Zookeeper + versions 3.4.0 or later). + + This describes how to set up HBase to mutually + authenticate with a ZooKeeper Quorum. ZooKeeper/HBase + mutual authentication (HBASE-2418) + is required as part of a complete secure HBase configuration + (HBASE-3025). + + For simplicity of explication, this section ignores + additional configuration required (Secure HDFS and Coprocessor + configuration). It's recommended to begin with an + HBase-managed Zookeeper configuration (as opposed to a + standalone Zookeeper quorum) for ease of learning. + + +
Operating System Prerequisites
+ + + You need to have a working Kerberos KDC setup. For + each $HOST that will run a ZooKeeper + server, you should have a principle + zookeeper/$HOST. For each such host, + add a service key (using the kadmin or + kadmin.local tool's ktadd + command) for zookeeper/$HOST and copy + this file to $HOST, and make it + readable only to the user that will run zookeeper on + $HOST. Note the location of this file, + which we will use below as + $PATH_TO_ZOOKEEPER_KEYTAB. + + + + Similarly, for each $HOST that will run + an HBase server (master or regionserver), you should + have a principle: hbase/$HOST. For each + host, add a keytab file called + hbase.keytab containing a service + key for hbase/$HOST, copy this file to + $HOST, and make it readable only to the + user that will run an HBase service on + $HOST. Note the location of this file, + which we will use below as + $PATH_TO_HBASE_KEYTAB. + + + + Each user who will be an HBase client should also be + given a Kerberos principal. This principal should + usually have a password assigned to it (as opposed to, + as with the HBase servers, a keytab file) which only + this user knows. The client's principal's + maxrenewlife should be set so that it can + be renewed enough so that the user can complete their + HBase client processes. For example, if a user runs a + long-running HBase client process that takes at most 3 + days, we might create this user's principal within + kadmin with: addprinc -maxrenewlife + 3days. The Zookeeper client and server + libraries manage their own ticket refreshment by + running threads that wake up periodically to do the + refreshment. + + + On each host that will run an HBase client + (e.g. hbase shell), add the following + file to the HBase home directory's conf + directory: + + + Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=false + useTicketCache=true; + }; + + + We'll refer to this JAAS configuration file as + $CLIENT_CONF below. + +
+ HBase-managed Zookeeper Configuration + + On each node that will run a zookeeper, a + master, or a regionserver, create a JAAS + configuration file in the conf directory of the node's + HBASE_HOME directory that looks like the + following: + + + Server { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + keyTab="$PATH_TO_ZOOKEEPER_KEYTAB" + storeKey=true + useTicketCache=false + principal="zookeeper/$HOST"; + }; + Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + useTicketCache=false + keyTab="$PATH_TO_HBASE_KEYTAB" + principal="hbase/$HOST"; + }; + + + where the $PATH_TO_HBASE_KEYTAB and + $PATH_TO_ZOOKEEPER_KEYTAB files are what + you created above, and $HOST is the hostname for that + node. + + The Server section will be used by + the Zookeeper quorum server, while the + Client section will be used by the HBase + master and regionservers. The path to this file should + be substituted for the text $HBASE_SERVER_CONF + in the hbase-env.sh + listing below. + + + The path to this file should be substituted for the + text $CLIENT_CONF in the + hbase-env.sh listing below. + + + Modify your hbase-env.sh to include the + following: + + + export HBASE_OPTS="-Djava.security.auth.login.config=$CLIENT_CONF" + export HBASE_MANAGES_ZK=true + export HBASE_ZOOKEEPER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + + + where $HBASE_SERVER_CONF and + $CLIENT_CONF are the full paths to the + JAAS configuration files created above. + + Modify your hbase-site.xml on each node + that will run zookeeper, master or regionserver to contain: + + + + hbase.zookeeper.quorum + $ZK_NODES + + + hbase.cluster.distributed + true + + + hbase.zookeeper.property.authProvider.1 + org.apache.zookeeper.server.auth.SASLAuthenticationProvider + + + hbase.zookeeper.property.kerberos.removeHostFromPrincipal + true + + + hbase.zookeeper.property.kerberos.removeRealmFromPrincipal + true + + + ]]> + + where $ZK_NODES is the + comma-separated list of hostnames of the Zookeeper + Quorum hosts. + + Start your hbase cluster by running one or more + of the following set of commands on the appropriate + hosts: + + + + bin/hbase zookeeper start + bin/hbase master start + bin/hbase regionserver start + + +
+ +
External Zookeeper Configuration + Add a JAAS configuration file that looks like: + + + Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + useTicketCache=false + keyTab="$PATH_TO_HBASE_KEYTAB" + principal="hbase/$HOST"; + }; + + + where the $PATH_TO_HBASE_KEYTAB is the keytab + created above for HBase services to run on this host, and $HOST is the + hostname for that node. Put this in the HBase home's + configuration directory. We'll refer to this file's + full pathname as $HBASE_SERVER_CONF below. + + Modify your hbase-env.sh to include the following: + + + export HBASE_OPTS="-Djava.security.auth.login.config=$CLIENT_CONF" + export HBASE_MANAGES_ZK=false + export HBASE_MASTER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + export HBASE_REGIONSERVER_OPTS="-Djava.security.auth.login.config=$HBASE_SERVER_CONF" + + + + Modify your hbase-site.xml on each node + that will run a master or regionserver to contain: + + + + hbase.zookeeper.quorum + $ZK_NODES + + + hbase.cluster.distributed + true + + + ]]> + + + where $ZK_NODES is the + comma-separated list of hostnames of the Zookeeper + Quorum hosts. + + + Add a zoo.cfg for each Zookeeper Quorum host containing: + + authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider + kerberos.removeHostFromPrincipal=true + kerberos.removeRealmFromPrincipal=true + + + Also on each of these hosts, create a JAAS configuration file containing: + + + Server { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + keyTab="$PATH_TO_ZOOKEEPER_KEYTAB" + storeKey=true + useTicketCache=false + principal="zookeeper/$HOST"; + }; + + + where $HOST is the hostname of each + Quorum host. We will refer to the full pathname of + this file as $ZK_SERVER_CONF below. + + + + + Start your Zookeepers on each Zookeeper Quorum host with: + + + SERVER_JVMFLAGS="-Djava.security.auth.login.config=$ZK_SERVER_CONF" bin/zkServer start + + + + + + Start your HBase cluster by running one or more of the following set of commands on the appropriate nodes: + + + + bin/hbase master start + bin/hbase regionserver start + + + +
+ +
+ Zookeeper Server Authentication Log Output + If the configuration above is successful, + you should see something similar to the following in + your Zookeeper server logs: + +11/12/05 22:43:39 INFO zookeeper.Login: successfully logged in. +11/12/05 22:43:39 INFO server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 +11/12/05 22:43:39 INFO zookeeper.Login: TGT refresh thread started. +11/12/05 22:43:39 INFO zookeeper.Login: TGT valid starting at: Mon Dec 05 22:43:39 UTC 2011 +11/12/05 22:43:39 INFO zookeeper.Login: TGT expires: Tue Dec 06 22:43:39 UTC 2011 +11/12/05 22:43:39 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:36:42 UTC 2011 +.. +11/12/05 22:43:59 INFO auth.SaslServerCallbackHandler: + Successfully authenticated client: authenticationID=hbase/ip-10-166-175-249.us-west-1.compute.internal@HADOOP.LOCALDOMAIN; + authorizationID=hbase/ip-10-166-175-249.us-west-1.compute.internal@HADOOP.LOCALDOMAIN. +11/12/05 22:43:59 INFO auth.SaslServerCallbackHandler: Setting authorizedID: hbase +11/12/05 22:43:59 INFO server.ZooKeeperServer: adding SASL authorization for authorizationID: hbase + + + + +
+ +
+ Zookeeper Client Authentication Log Output + On the Zookeeper client side (HBase master or regionserver), + you should see something similar to the following: + + +11/12/05 22:43:59 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-166-175-249.us-west-1.compute.internal:2181 sessionTimeout=180000 watcher=master:60000 +11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.166.175.249:2181 +11/12/05 22:43:59 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 14851@ip-10-166-175-249 +11/12/05 22:43:59 INFO zookeeper.Login: successfully logged in. +11/12/05 22:43:59 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. +11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh thread started. +11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, initiating session +11/12/05 22:43:59 INFO zookeeper.Login: TGT valid starting at: Mon Dec 05 22:43:59 UTC 2011 +11/12/05 22:43:59 INFO zookeeper.Login: TGT expires: Tue Dec 06 22:43:59 UTC 2011 +11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:30:37 UTC 2011 +11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, sessionid = 0x134106594320000, negotiated timeout = 180000 + + +
+ +
+ Configuration from Scratch + + This has been tested on the current standard Amazon + Linux AMI. First setup KDC and principals as + described above. Next checkout code and run a sanity + check. + + + git clone git://git.apache.org/hbase.git + cd hbase + mvn -Psecurity,localTests clean test -Dtest=TestZooKeeperACL + + + Then configure HBase as described above. + Manually edit target/cached_classpath.txt (see below).. + + + bin/hbase zookeeper & + bin/hbase master & + bin/hbase regionserver & + +
+ + +
+ Future improvements + +
Fix target/cached_classpath.txt + + You must override the standard hadoop-core jar file from the + target/cached_classpath.txt + file with the version containing the HADOOP-7070 fix. You can use the following script to do this: + + + echo `find ~/.m2 -name "*hadoop-core*7070*SNAPSHOT.jar"` ':' `cat target/cached_classpath.txt` | sed 's/ //g' > target/tmp.txt + mv target/tmp.txt target/cached_classpath.txt + + + + +
+ +
+ Set JAAS configuration + programmatically + + + This would avoid the need for a separate Hadoop jar + that fixes HADOOP-7070. +
+ +
+ Elimination of + <code>kerberos.removeHostFromPrincipal</code> and + <code>kerberos.removeRealmFromPrincipal</code> +
+ +
+ + +
+ + + + +