Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HBASE-2692 Master rewrite and cleanup for 0.90 

Patch brought over from 0.90_master_rewrite branch.

Replication test is broke as are some of the rest tests.
Others should be passing.

Some of the changes made in this fat patch:

+ In HLogKey, we now use encoded region name instead of full region name.
+ On split, daughters are opened on the parent's regionserver; let the new balancer
sort them out later when it cuts in.
+ Added move region from one server to another as well as enable/disable balancer.
+ All .META. and -ROOT- edits go via new *Editor and *Reader classes -- no more
do we have 5 different ways of reading and editing .META.
+ Rather than 3 different listeners to hlog each w/ own way of listening, instead
we only have WALObserver now.
+ New Server Interface that has whats common to HMaster and RegionServer. Also
new Services Interface.  This should make test writing cleaner making it so
less need of full cluster context testing anything -- e.g. the new
Interfaces are good w/ Mockito.
+ New balacner that runs on a period and takes into consideration all load
across cluster.
+ Table online/offline is now a flag in ZK; the offline flag on a region is
just used splitting from here on out.
+ Moved fixup of failed add of daughter edits to .META. into shutdown server
recover code (It used to be in basescanner).
+ The heartbeat now sends master the regionserver load and is used sending
shutdown message from master to regionserver ONLY; all other messages are
via zk (HMsg is pretty bare now).
+ No more Worker in RS and ToDoQueue in master.  Both in master and regionserver
we use handlers instead run out of Executors.
+ Client can not send split, flush, compact direct to RS; no longer does
it go via master.
+ Server shutdown runs differently now. All are watching a flag in zk.
When RS notices its gone, it closes all user-space regions. If thats all
it was carrying, then it goes down.  Otherwise, waits on master to send
the shutdown msg via heartbeat.



git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@991397 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Michael Stack 2010-08-31 23:51:44 +00:00
parent 8586a6238a
commit f521309532
219 changed files with 15099 additions and 13508 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
bin/rolling-restart.sh
View File

@ -49,7 +49,7 @@ then
fi
# quick function to get a value from the HBase config file
distMode=`$bin/hbase org.apache.hadoop.hbase.HBaseConfTool hbase.cluster.distributed`
distMode=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed`
if [ "$distMode" == 'false' ]; then
"$bin"/hbase-daemon.sh restart master
else
@ -59,9 +59,9 @@ else
--hosts "${HBASE_BACKUP_MASTERS}" stop master-backup
# make sure the master znode has been deleted before continuing
zparent=`$bin/hbase org.apache.hadoop.hbase.HBaseConfTool zookeeper.znode.parent`
zparent=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool zookeeper.znode.parent`
if [ "$zparent" == "null" ]; then zparent="/hbase"; fi
zmaster=`$bin/hbase org.apache.hadoop.hbase.HBaseConfTool zookeeper.znode.master`
zmaster=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool zookeeper.znode.master`
if [ "$zmaster" == "null" ]; then zmaster="master"; fi
zmaster=$zparent/$zmaster
echo -n "Waiting for Master ZNode to expire"
@ -134,7 +134,7 @@ then
fi
# quick function to get a value from the HBase config file
distMode=`$bin/hbase org.apache.hadoop.hbase.HBaseConfTool hbase.cluster.distributed`
distMode=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed`
if [ "$distMode" == 'false' ]; then
"$bin"/hbase-daemon.sh restart master
else
@ -144,9 +144,9 @@ else
--hosts "${HBASE_BACKUP_MASTERS}" stop master-backup
# make sure the master znode has been deleted before continuing
zparent=`$bin/hbase org.apache.hadoop.hbase.HBaseConfTool zookeeper.znode.parent`
zparent=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool zookeeper.znode.parent`
if [ "$zparent" == "null" ]; then zparent="/hbase"; fi
zmaster=`$bin/hbase org.apache.hadoop.hbase.HBaseConfTool zookeeper.znode.master`
zmaster=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool zookeeper.znode.master`
if [ "$zmaster" == "null" ]; then zmaster="master"; fi
zmaster=$zparent/$zmaster
echo -n "Waiting for Master ZNode to expire"

2
bin/start-hbase.sh
View File

@ -38,7 +38,7 @@ then
exit $errCode
fi
distMode=`$bin/hbase org.apache.hadoop.hbase.HBaseConfTool hbase.cluster.distributed`
distMode=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed`
if [ "$distMode" == 'false' ]

2
bin/stop-hbase.sh
View File

@ -57,7 +57,7 @@ while kill -0 `cat $pid` > /dev/null 2>&1; do
done
# distributed == false means that the HMaster will kill ZK when it exits
distMode=`$bin/hbase org.apache.hadoop.hbase.HBaseConfTool hbase.cluster.distributed`
distMode=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed`
if [ "$distMode" == 'true' ]
then
# TODO: store backup masters in ZooKeeper and have the primary send them a shutdown message

2
conf/hbase-site.xml
View File

@ -2,7 +2,7 @@
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Copyright 2009 The Apache Software Foundation
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file

77
conf/hbase-site.xml.psuedo-distributed.template
View File

@ -1,77 +0,0 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Copyright 2009 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<!-- NEEDED WHETHER OR NOT YOU ARE RUNNING OVER HDFS -->
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>For psuedo-distributed, you want to set this to true.
false means that HBase tries to put Master + RegionServers in one process.
Pseudo-distributed = seperate processes/pids</description>
</property> <property>
<name>hbase.regionserver.hlog.replication</name>
<value>1</value>
<description>For HBase to offer good data durability, we roll logs if
filesystem replication falls below a certain amount. In psuedo-distributed
mode, you normally only have the local filesystem or 1 HDFS DataNode, so you
don't want to roll logs constantly.</description>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/tmp/hbase-testing</value>
<description>Temporary directory on the local filesystem.</description>
</property>
<!-- DEFAULT = use local filesystem, not HDFS
ADD THESE LINES if you have a copy of HDFS source and want to run HBase
psuedo-distributed over a psuedo-distributed HDFS cluster.
For HDFS psuedo-distributed setup, see their documentation:
http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#PseudoDistributed
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase-testing</value>
<description>The directory shared by region servers.
Should be fully-qualified to include the filesystem to use.
E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR
</description>
</property>
-->
<!-- OPTIONAL: You might want to add these options depending upon your use case
<property>
<name>dfs.support.append</name>
<value>true</value>
<description>Allow append support (if you want to test data durability with HDFS)
</description>
</property>
-->
</configuration>

11
pom.xml
View File

@ -782,11 +782,11 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
</dependencies>
<!--
@ -961,6 +961,7 @@
<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<version>0.6</version>
</plugin>
</plugins>
</reporting>

32
src/main/java/org/apache/hadoop/hbase/ValueOverMaxLengthException.java → src/main/java/org/apache/hadoop/hbase/Abortable.java
View File

@ -1,5 +1,5 @@
/**
* Copyright 2008 The Apache Software Foundation
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
@ -20,24 +20,18 @@
package org.apache.hadoop.hbase;
/**
* Thrown when a value is longer than the specified LENGTH
* Interface to support the aborting of a given server or client.
* <p>
* This is used primarily for ZooKeeper usage when we could get an unexpected
* and fatal exception, requiring an abort.
* <p>
* Implemented by the Master, RegionServer, and TableServers (client).
*/
public class ValueOverMaxLengthException extends DoNotRetryIOException {
private static final long serialVersionUID = -5525656352372008316L;
public interface Abortable {
/**
* default constructor
* Abort the server or client.
* @param why Why we're aborting.
* @param e Throwable that caused abort. Can be null.
*/
public ValueOverMaxLengthException() {
super();
}
/**
* @param message
*/
public ValueOverMaxLengthException(String message) {
super(message);
}
}
public void abort(String why, Throwable e);
}

23
src/main/java/org/apache/hadoop/hbase/Chore.java
View File

@ -19,8 +19,6 @@
*/
package org.apache.hadoop.hbase;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.util.Sleeper;
@ -32,24 +30,24 @@ import org.apache.hadoop.hbase.util.Sleeper;
* Implementers just need to add checking if there is work to be done and if
* so, do it. Its the base of most of the chore threads in hbase.
*
* Don't subclass Chore if the task relies on being woken up for something to
* <p>Don't subclass Chore if the task relies on being woken up for something to
* do, such as an entry being added to a queue, etc.
*/
public abstract class Chore extends Thread {
private final Log LOG = LogFactory.getLog(this.getClass());
private final Sleeper sleeper;
protected volatile AtomicBoolean stop;
protected final Stoppable stopper;
/**
* @param p Period at which we should run. Will be adjusted appropriately
* should we find work and it takes time to complete.
* @param s When this flag is set to true, this thread will cleanup and exit
* cleanly.
* @param stopper When {@link Stoppable#isStopped()} is true, this thread will
* cleanup and exit cleanly.
*/
public Chore(String name, final int p, final AtomicBoolean s) {
public Chore(String name, final int p, final Stoppable stopper) {
super(name);
this.sleeper = new Sleeper(p, s);
this.stop = s;
this.sleeper = new Sleeper(p, stopper);
this.stopper = stopper;
}
/**
@ -59,7 +57,7 @@ public abstract class Chore extends Thread {
public void run() {
try {
boolean initialChoreComplete = false;
while (!this.stop.get()) {
while (!this.stopper.isStopped()) {
long startTime = System.currentTimeMillis();
try {
if (!initialChoreComplete) {
@ -69,15 +67,14 @@ public abstract class Chore extends Thread {
}
} catch (Exception e) {
LOG.error("Caught exception", e);
if (this.stop.get()) {
if (this.stopper.isStopped()) {
continue;
}
}
this.sleeper.sleep(startTime);
}
} catch (Throwable t) {
LOG.fatal("Caught error. Starting shutdown.", t);
this.stop.set(true);
LOG.fatal(getName() + "error", t);
} finally {
LOG.info(getName() + " exiting");
}

19
src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
View File

@ -27,9 +27,9 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import java.util.NavigableMap;
import java.util.TreeMap;
import org.apache.hadoop.hbase.master.AssignmentManager.RegionState;
import org.apache.hadoop.io.VersionedWritable;
/**
@ -53,7 +53,7 @@ public class ClusterStatus extends VersionedWritable {
private String hbaseVersion;
private Collection<HServerInfo> liveServerInfo;
private Collection<String> deadServers;
private NavigableMap<String, String> intransition;
private Map<String, RegionState> intransition;
/**
* Constructor, for Writable
@ -186,11 +186,11 @@ public class ClusterStatus extends VersionedWritable {
this.deadServers = deadServers;
}
public Map<String, String> getRegionsInTransition() {
public Map<String, RegionState> getRegionsInTransition() {
return this.intransition;
}
public void setRegionsInTransition(final NavigableMap<String, String> m) {
public void setRegionsInTransition(final Map<String, RegionState> m) {
this.intransition = m;
}
@ -210,9 +210,9 @@ public class ClusterStatus extends VersionedWritable {
out.writeUTF(server);
}
out.writeInt(this.intransition.size());
for (Map.Entry<String, String> e: this.intransition.entrySet()) {
for (Map.Entry<String, RegionState> e: this.intransition.entrySet()) {
out.writeUTF(e.getKey());
out.writeUTF(e.getValue());
e.getValue().write(out);
}
}
@ -232,11 +232,12 @@ public class ClusterStatus extends VersionedWritable {
deadServers.add(in.readUTF());
}
count = in.readInt();
this.intransition = new TreeMap<String, String>();
this.intransition = new TreeMap<String, RegionState>();
for (int i = 0; i < count; i++) {
String key = in.readUTF();
String value = in.readUTF();
this.intransition.put(key, value);
RegionState regionState = new RegionState();
regionState.readFields(in);
this.intransition.put(key, regionState);
}
}
}

2
src/main/java/org/apache/hadoop/hbase/DroppedSnapshotException.java
View File

@ -38,4 +38,4 @@ public class DroppedSnapshotException extends IOException {
public DroppedSnapshotException() {
super();
}
}
}

15
src/main/java/org/apache/hadoop/hbase/HConstants.java
View File

@ -132,6 +132,15 @@ public final class HConstants {
/** Parameter name for how often threads should wake up */
public static final String THREAD_WAKE_FREQUENCY = "hbase.server.thread.wakefrequency";
/** Default value for thread wake frequency */
public static final int DEFAULT_THREAD_WAKE_FREQUENCY = 10 * 1000;
/** Number of retries for the client */
public static final String NUM_CLIENT_RETRIES = "hbase.client.retries.number";
/** Default number of retries for the client */
public static final int DEFAULT_NUM_CLIENT_RETRIES = 2;
/** Parameter name for how often a region should should perform a major compaction */
public static final String MAJOR_COMPACTION_PERIOD = "hbase.hregion.majorcompaction";
@ -197,9 +206,6 @@ public final class HConstants {
/** The catalog family */
public static final byte [] CATALOG_FAMILY = Bytes.toBytes(CATALOG_FAMILY_STR);
/** The catalog historian family */
public static final byte [] CATALOG_HISTORIAN_FAMILY = Bytes.toBytes("historian");
/** The regioninfo column qualifier */
public static final byte [] REGIONINFO_QUALIFIER = Bytes.toBytes("regioninfo");
@ -343,7 +349,8 @@ public final class HConstants {
* HRegion server lease period in milliseconds. Clients must report in within this period
* else they are considered dead. Unit measured in ms (milliseconds).
*/
public static String HBASE_REGIONSERVER_LEASE_PERIOD_KEY = "hbase.regionserver.lease.period";
public static String HBASE_REGIONSERVER_LEASE_PERIOD_KEY =
"hbase.regionserver.lease.period";
/**

106
src/main/java/org/apache/hadoop/hbase/HMsg.java
View File

@ -27,106 +27,37 @@ import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.io.Writable;
/**
* HMsg is for communicating instructions between the HMaster and the
* HRegionServers.
* HMsg is used to send messages between master and regionservers. Messages are
* sent as payload on the regionserver-to-master heartbeats. Region assignment
* does not use this mechanism. It goes via zookeeper.
*
* Most of the time the messages are simple but some messages are accompanied
* by the region affected. HMsg may also carry optional message.
* <p>Most of the time the messages are simple but some messages are accompanied
* by the region affected. HMsg may also carry an optional message.
*
* <p>TODO: Clean out all messages that go from master to regionserver; by
* design, these are to go via zk from here on out.
*/
public class HMsg implements Writable {
public static final HMsg REGIONSERVER_QUIESCE =
new HMsg(Type.MSG_REGIONSERVER_QUIESCE);
public static final HMsg REGIONSERVER_STOP =
new HMsg(Type.MSG_REGIONSERVER_STOP);
public static final HMsg [] STOP_REGIONSERVER_ARRAY =
new HMsg [] {new HMsg(Type.STOP_REGIONSERVER)};
public static final HMsg [] EMPTY_HMSG_ARRAY = new HMsg[0];
/**
* Message types sent between master and regionservers
*/
public static enum Type {
/** null message */
MSG_NONE,
// Message types sent from master to region server
/** Start serving the specified region */
MSG_REGION_OPEN,
/** Stop serving the specified region */
MSG_REGION_CLOSE,
/** Split the specified region */
MSG_REGION_SPLIT,
/** Compact the specified region */
MSG_REGION_COMPACT,
/** Master tells region server to stop */
MSG_REGIONSERVER_STOP,
/** Stop serving the specified region and don't report back that it's
* closed
/** Master tells region server to stop.
*/
MSG_REGION_CLOSE_WITHOUT_REPORT,
/** Stop serving user regions */
MSG_REGIONSERVER_QUIESCE,
// Message types sent from the region server to the master
/** region server is now serving the specified region */
MSG_REPORT_OPEN,
/** region server is no longer serving the specified region */
MSG_REPORT_CLOSE,
/** region server is processing open request */
MSG_REPORT_PROCESS_OPEN,
STOP_REGIONSERVER,
/**
* Region server split the region associated with this message.
*
* Note that this message is immediately followed by two MSG_REPORT_OPEN
* messages, one for each of the new regions resulting from the split
* @deprecated See MSG_REPORT_SPLIT_INCLUDES_DAUGHTERS
*/
MSG_REPORT_SPLIT,
/**
* Region server is shutting down
*
* Note that this message is followed by MSG_REPORT_CLOSE messages for each
* region the region server was serving, unless it was told to quiesce.
*/
MSG_REPORT_EXITING,
/** Region server has closed all user regions but is still serving meta
* regions
*/
MSG_REPORT_QUIESCED,
/**
* Flush
*/
MSG_REGION_FLUSH,
/**
* Run Major Compaction
*/
MSG_REGION_MAJOR_COMPACT,
/**
* Region server split the region associated with this message.
*
* Its like MSG_REPORT_SPLIT only it carries the daughters in the message
* rather than send them individually in MSG_REPORT_OPEN messages.
*/
MSG_REPORT_SPLIT_INCLUDES_DAUGHTERS,
REGION_SPLIT,
/**
* When RegionServer receives this message, it goes into a sleep that only
* an exit will cure. This message is sent by unit tests simulating
* pathological states.
*/
TESTING_MSG_BLOCK_RS,
TESTING_BLOCK_REGIONSERVER,
}
private Type type = null;
@ -137,7 +68,7 @@ public class HMsg implements Writable {
/** Default constructor. Used during deserialization */
public HMsg() {
this(Type.MSG_NONE);
this(null);
}
/**
@ -181,9 +112,6 @@ public class HMsg implements Writable {
*/
public HMsg(final HMsg.Type type, final HRegionInfo hri,
final HRegionInfo daughterA, final HRegionInfo daughterB, final byte[] msg) {
if (type == null) {
throw new NullPointerException("Message type cannot be null");
}
this.type = type;
if (hri == null) {
throw new NullPointerException("Region cannot be null");
@ -301,7 +229,7 @@ public class HMsg implements Writable {
out.writeBoolean(true);
Bytes.writeByteArray(out, this.message);
}
if (this.type.equals(Type.MSG_REPORT_SPLIT_INCLUDES_DAUGHTERS)) {
if (this.type.equals(Type.REGION_SPLIT)) {
this.daughterA.write(out);
this.daughterB.write(out);
}
@ -318,7 +246,7 @@ public class HMsg implements Writable {
if (hasMessage) {
this.message = Bytes.readByteArray(in);
}
if (this.type.equals(Type.MSG_REPORT_SPLIT_INCLUDES_DAUGHTERS)) {
if (this.type.equals(Type.REGION_SPLIT)) {
this.daughterA = new HRegionInfo();
this.daughterB = new HRegionInfo();
this.daughterA.readFields(in);

45
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
View File

@ -103,13 +103,28 @@ public class HRegionInfo extends VersionedWritable implements WritableComparable
// old format region name. ROOT and first META region also
// use this format.EncodedName is the JenkinsHash value.
int hashVal = Math.abs(JenkinsHash.getInstance().hash(regionName,
regionName.length,
0));
regionName.length, 0));
encodedName = String.valueOf(hashVal);
}
return encodedName;
}
/**
* Use logging.
* @param encodedRegionName The encoded regionname.
* @return <code>-ROOT-</code> if passed <code>70236052</code> or
* <code>.META.</code> if passed </code>1028785192</code> else returns
* <code>encodedRegionName</code>
*/
public static String prettyPrint(final String encodedRegionName) {
if (encodedRegionName.equals("70236052")) {
return encodedRegionName + "/-ROOT-";
} else if (encodedRegionName.equals("1028785192")) {
return encodedRegionName + "/.META.";
}
return encodedRegionName;
}
/** delimiter used between portions of a region name */
public static final int DELIMITER = ',';
@ -133,6 +148,7 @@ public class HRegionInfo extends VersionedWritable implements WritableComparable
//TODO: Move NO_HASH to HStoreFile which is really the only place it is used.
public static final String NO_HASH = null;
private volatile String encodedName = NO_HASH;
private byte [] encodedNameAsBytes = null;
private void setHashCode() {
int result = Arrays.hashCode(this.regionName);
@ -315,6 +331,24 @@ public class HRegionInfo extends VersionedWritable implements WritableComparable
return b;
}
/**
* Gets the table name from the specified region name.
* @param regionName
* @return
*/
public static byte [] getTableName(byte [] regionName) {
int offset = -1;
for (int i = 0; i < regionName.length; i++) {
if (regionName[i] == DELIMITER) {
offset = i;
break;
}
}
byte [] tableName = new byte[offset];
System.arraycopy(regionName, 0, tableName, 0, offset);
return tableName;
}
/**
* Separate elements of a regionName.
* @param regionName
@ -393,6 +427,13 @@ public class HRegionInfo extends VersionedWritable implements WritableComparable
return this.encodedName;
}
public synchronized byte [] getEncodedNameAsBytes() {
if (this.encodedNameAsBytes == null) {
this.encodedNameAsBytes = Bytes.toBytes(getEncodedName());
}
return this.encodedNameAsBytes;
}
/** @return the startKey */
public byte [] getStartKey(){
return startKey;

1
src/main/java/org/apache/hadoop/hbase/HRegionLocation.java
View File

@ -24,6 +24,7 @@ package org.apache.hadoop.hbase;
* HRegionServer serving the region
*/
public class HRegionLocation implements Comparable<HRegionLocation> {
// TODO: Is this class necessary? Why not just have a Pair?
private HRegionInfo regionInfo;
private HServerAddress serverAddress;

2
src/main/java/org/apache/hadoop/hbase/HServerAddress.java
View File

@ -46,7 +46,7 @@ public class HServerAddress implements WritableComparable<HServerAddress> {
*/
public HServerAddress(InetSocketAddress address) {
this.address = address;
this.stringValue = address.getAddress().getHostAddress() + ":" +
this.stringValue = address.getAddress().getHostName() + ":" +
address.getPort();
checkBindAddressCanBeResolved();
}

14
src/main/java/org/apache/hadoop/hbase/HServerInfo.java
View File

@ -23,6 +23,7 @@ import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.Comparator;
import java.util.Set;
import org.apache.hadoop.hbase.regionserver.HRegionServer;
@ -147,6 +148,8 @@ public class HServerInfo implements WritableComparable<HServerInfo> {
}
/**
* Gets the unique server instance name. Includes the hostname, port, and
* start code.
* @return Server name made of the concatenation of hostname, port and
* startcode formatted as <code>&lt;hostname> ',' &lt;port> ',' &lt;startcode></code>
*/
@ -242,6 +245,17 @@ public class HServerInfo implements WritableComparable<HServerInfo> {
return this.getServerName().compareTo(o.getServerName());
}
/**
* Orders HServerInfos by load then name. Natural/ascending order.
*/
public static class LoadComparator implements Comparator<HServerInfo> {
@Override
public int compare(HServerInfo left, HServerInfo right) {
int loadCompare = left.getLoad().compareTo(right.getLoad());
return loadCompare != 0 ? loadCompare : left.compareTo(right);
}
}
/**
* Utility method that does a find of a servername or a hostandport combination
* in the passed Set.

5
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
View File

@ -678,10 +678,5 @@ public class HTableDescriptor implements WritableComparable<HTableDescriptor> {
10, // Ten is arbitrary number. Keep versions to help debuggging.
Compression.Algorithm.NONE.getName(), true, true, 8 * 1024,
HConstants.FOREVER, StoreFile.BloomType.NONE.toString(),
HConstants.REPLICATION_SCOPE_LOCAL),
new HColumnDescriptor(HConstants.CATALOG_HISTORIAN_FAMILY,
HConstants.ALL_VERSIONS, Compression.Algorithm.NONE.getName(),
false, false, 8 * 1024,
HConstants.WEEK_IN_SECONDS,StoreFile.BloomType.NONE.toString(),
HConstants.REPLICATION_SCOPE_LOCAL)});
}

50
src/main/java/org/apache/hadoop/hbase/InvalidFamilyOperationException.java Normal file
View File

@ -0,0 +1,50 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase;
import java.io.IOException;
/**
* Thrown if a request is table schema modification is requested but
* made for an invalid family name.
*/
public class InvalidFamilyOperationException extends IOException {
private static final long serialVersionUID = 1L << 22 - 1L;
/** default constructor */
public InvalidFamilyOperationException() {
super();
}
/**
* Constructor
* @param s message
*/
public InvalidFamilyOperationException(String s) {
super(s);
}
/**
* Constructor taking another exception.
* @param e Exception to grab data from.
*/
public InvalidFamilyOperationException(Exception e) {
super(e);
}
}

91
src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java Normal file
View File

@ -0,0 +1,91 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher;
/**
* Manages the location of the current active Master for this RegionServer.
* <p>
* Listens for ZooKeeper events related to the master address. The node
* <code>/master</code> will contain the address of the current master.
* This listener is interested in
* <code>NodeDeleted</code> and <code>NodeCreated</code> events on
* <code>/master</code>.
* <p>
* Utilizes {@link ZooKeeperNodeTracker} for zk interactions.
* <p>
* You can get the current master via {@link #getMasterAddress()} or the
* blocking method {@link #waitMasterAddress()}.
*/
public class MasterAddressTracker extends ZooKeeperNodeTracker {
/**
* Construct a master address listener with the specified
* <code>zookeeper</code> reference.
* <p>
* This constructor does not trigger any actions, you must call methods
* explicitly. Normally you will just want to execute {@link #start()} to
* begin tracking of the master address.
*
* @param watcher zk reference and watcher
* @param abortable abortable in case of fatal error
*/
public MasterAddressTracker(ZooKeeperWatcher watcher, Abortable abortable) {
super(watcher, watcher.masterAddressZNode, abortable);
}
/**
* Get the address of the current master if one is available. Returns null
* if no current master.
*
* Use {@link #waitMasterAddress} if you want to block until the master is
* available.
* @return server address of current active master, or null if none available
*/
public HServerAddress getMasterAddress() {
byte [] data = super.getData();
return data == null ? null : new HServerAddress(Bytes.toString(data));
}
/**
* Check if there is a master available.
* @return true if there is a master set, false if not.
*/
public boolean hasMaster() {
return super.getData() != null;
}
/**
* Get the address of the current master. If no master is available, method
* will block until one is available, the thread is interrupted, or timeout
* has passed.
*
* @param timeout maximum time to wait for master in millis, 0 for forever
* @return server address of current active master, null if timed out
* @throws InterruptedException if the thread is interrupted while waiting
*/
public synchronized HServerAddress waitForMaster(long timeout)
throws InterruptedException {
byte [] data = super.blockUntilAvailable();
return data == null ? null : new HServerAddress(Bytes.toString(data));
}
}

6
src/main/java/org/apache/hadoop/hbase/master/NotAllMetaRegionsOnlineException.java → src/main/java/org/apache/hadoop/hbase/NotAllMetaRegionsOnlineException.java
View File

@ -18,7 +18,7 @@
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
package org.apache.hadoop.hbase;
import org.apache.hadoop.hbase.DoNotRetryIOException;
@ -27,7 +27,6 @@ import org.apache.hadoop.hbase.DoNotRetryIOException;
*/
public class NotAllMetaRegionsOnlineException extends DoNotRetryIOException {
private static final long serialVersionUID = 6439786157874827523L;
/**
* default constructor
*/
@ -41,5 +40,4 @@ public class NotAllMetaRegionsOnlineException extends DoNotRetryIOException {
public NotAllMetaRegionsOnlineException(String message) {
super(message);
}
}
}

4
src/main/java/org/apache/hadoop/hbase/PleaseHoldException.java
View File

@ -26,9 +26,9 @@ import java.io.IOException;
* and restarted so fast that the master still hasn't processed the server
* shutdown of the first instance.
*/
@SuppressWarnings("serial")
public class PleaseHoldException extends IOException {
public PleaseHoldException(String message) {
super(message);
}
}
}

54
src/main/java/org/apache/hadoop/hbase/Server.java Normal file
View File

@ -0,0 +1,54 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.catalog.CatalogTracker;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher;
/**
* Defines the set of shared functions implemented by HBase servers (Masters
* and RegionServers).
*/
public interface Server extends Abortable, Stoppable {
/**
* Gets the configuration object for this server.
*/
public Configuration getConfiguration();
/**
* Gets the ZooKeeper instance for this server.
*/
public ZooKeeperWatcher getZooKeeper();
/**
* @return Master's instance of {@link CatalogTracker}
*/
public CatalogTracker getCatalogTracker();
/**
* Gets the unique server name for this server.
* If a RegionServer, it returns a concatenation of hostname, port and
* startcode formatted as <code>&lt;hostname> ',' &lt;port> ',' &lt;startcode></code>.
* If the master, it returns <code>&lt;hostname> ':' &lt;port>'.
* @return unique server name
*/
public String getServerName();
}

21
src/main/java/org/apache/hadoop/hbase/WritableComparator.java → src/main/java/org/apache/hadoop/hbase/Stoppable.java
View File

@ -1,5 +1,5 @@
/**
* Copyright 2008 The Apache Software Foundation
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
@ -19,11 +19,18 @@
*/
package org.apache.hadoop.hbase;
import java.util.Comparator;
import org.apache.hadoop.io.Writable;
/**
* Interface that brings writable and comparable together
* Implementers are Stoppable.
*/
public interface WritableComparator<T> extends Writable, Comparator<T> {}
public interface Stoppable {
/**
* Stop this service.
* @param why Why we're stopping.
*/
public void stop(String why);
/**
* @return True if {@link #stop(String)} has been closed.
*/
public boolean isStopped();
}

23
src/main/java/org/apache/hadoop/hbase/ColumnNameParseException.java → src/main/java/org/apache/hadoop/hbase/UnknownRegionException.java
View File

@ -1,5 +1,5 @@
/**
* Copyright 2008 The Apache Software Foundation
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
@ -19,22 +19,15 @@
*/
package org.apache.hadoop.hbase;
import java.io.IOException;
/**
* Thrown if issue with passed column name.
* Thrown when we are asked to operate on a region we know nothing about.
*/
public class ColumnNameParseException extends DoNotRetryIOException {
public class UnknownRegionException extends IOException {
private static final long serialVersionUID = 1968858760475205392L;
private static final long serialVersionUID = -2897373353949942302L;
/** default constructor */
public ColumnNameParseException() {
super();
}
/**
* @param message
*/
public ColumnNameParseException(String message) {
super(message);
public UnknownRegionException(String regionName) {
super(regionName);
}
}

2
src/main/java/org/apache/hadoop/hbase/UnknownRowLockException.java
View File

@ -38,4 +38,4 @@ public class UnknownRowLockException extends DoNotRetryIOException {
public UnknownRowLockException(String s) {
super(s);
}
}
}

4
src/main/java/org/apache/hadoop/hbase/YouAreDeadException.java
View File

@ -26,9 +26,9 @@ import java.io.IOException;
* already being processed as dead. This can happen when a region server loses
* its session but didn't figure it yet.
*/
@SuppressWarnings("serial")
public class YouAreDeadException extends IOException {
public YouAreDeadException(String message) {
super(message);
}
}
}

25
src/main/java/org/apache/hadoop/hbase/master/InvalidColumnNameException.java → src/main/java/org/apache/hadoop/hbase/ZooKeeperConnectionException.java
View File

@ -17,18 +17,17 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.DoNotRetryIOException;
package org.apache.hadoop.hbase;
import java.io.IOException;
/**
* Thrown when an invalid column name is encountered
* Thrown if the client can't connect to zookeeper
*/
public class InvalidColumnNameException extends DoNotRetryIOException {
private static final long serialVersionUID = 1L << 29 - 1L;
public class ZooKeeperConnectionException extends IOException {
private static final long serialVersionUID = 1L << 23 - 1L;
/** default constructor */
public InvalidColumnNameException() {
public ZooKeeperConnectionException() {
super();
}
@ -36,7 +35,15 @@ public class InvalidColumnNameException extends DoNotRetryIOException {
* Constructor
* @param s message
*/
public InvalidColumnNameException(String s) {
public ZooKeeperConnectionException(String s) {
super(s);
}
}
/**
* Constructor taking another exception.
* @param e Exception to grab data from.
*/
public ZooKeeperConnectionException(Exception e) {
super(e);
}
}

38
src/main/java/org/apache/hadoop/hbase/avro/AvroServer.java
View File

@ -20,10 +20,7 @@ package org.apache.hadoop.hbase.avro;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericArray;
@ -33,33 +30,17 @@ import org.apache.avro.specific.SpecificResponder;
import org.apache.avro.util.Utf8;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.TableExistsException;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.avro.generated.AClusterStatus;
import org.apache.hadoop.hbase.avro.generated.AColumnValue;
import org.apache.hadoop.hbase.avro.generated.ACompressionAlgorithm;
import org.apache.hadoop.hbase.avro.generated.ADelete;
import org.apache.hadoop.hbase.avro.generated.AFamilyDescriptor;
import org.apache.hadoop.hbase.avro.generated.AGet;
import org.apache.hadoop.hbase.avro.generated.AIllegalArgument;
import org.apache.hadoop.hbase.avro.generated.AIOError;
import org.apache.hadoop.hbase.avro.generated.AIllegalArgument;
import org.apache.hadoop.hbase.avro.generated.AMasterNotRunning;
import org.apache.hadoop.hbase.avro.generated.APut;
import org.apache.hadoop.hbase.avro.generated.AResult;
@ -67,6 +48,12 @@ import org.apache.hadoop.hbase.avro.generated.AScan;
import org.apache.hadoop.hbase.avro.generated.ATableDescriptor;
import org.apache.hadoop.hbase.avro.generated.ATableExists;
import org.apache.hadoop.hbase.avro.generated.HBase;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;
/**
* Start an Avro server
@ -135,10 +122,9 @@ public class AvroServer {
// TODO(hammer): figure out appropriate setting of maxSize for htablePool
/**
* Constructs an HBaseImpl object.
*
* @throws MasterNotRunningException
* @throws IOException
*/
HBaseImpl() throws MasterNotRunningException {
HBaseImpl() throws IOException {
conf = HBaseConfiguration.create();
admin = new HBaseAdmin(conf);
htablePool = new HTablePool(conf, 10);
@ -376,8 +362,7 @@ public class AvroServer {
// NB: Asynchronous operation
public Void modifyFamily(ByteBuffer table, ByteBuffer familyName, AFamilyDescriptor familyDescriptor) throws AIOError {
try {
admin.modifyColumn(Bytes.toBytes(table), Bytes.toBytes(familyName),
AvroUtil.afdToHCD(familyDescriptor));
admin.modifyColumn(Bytes.toBytes(table), AvroUtil.afdToHCD(familyDescriptor));
return null;
} catch (IOException e) {
AIOError ioe = new AIOError();
@ -506,7 +491,6 @@ public class AvroServer {
aie.message = new Utf8("scanner ID is invalid: " + scannerId);
throw aie;
}
Result[] results = null;
return AvroUtil.resultsToAResults(scanner.next(numberOfRows));
} catch (IOException e) {
AIOError ioe = new AIOError();
@ -566,7 +550,7 @@ public class AvroServer {
Log LOG = LogFactory.getLog("AvroServer");
LOG.info("starting HBase Avro server on port " + Integer.toString(port));
SpecificResponder r = new SpecificResponder(HBase.class, new HBaseImpl());
HttpServer server = new HttpServer(r, 9090);
new HttpServer(r, 9090);
Thread.sleep(1000000);
}

395
src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java Normal file
View File

@ -0,0 +1,395 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.catalog;
import java.io.IOException;
import java.net.ConnectException;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.Abortable;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException;
import org.apache.hadoop.hbase.NotServingRegionException;
import org.apache.hadoop.hbase.client.RetriesExhaustedException;
import org.apache.hadoop.hbase.client.ServerConnection;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.zookeeper.MetaNodeTracker;
import org.apache.hadoop.hbase.zookeeper.RootRegionTracker;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher;
import org.apache.zookeeper.KeeperException;
/**
* Tracks the availability of the catalog tables <code>-ROOT-</code> and
* <code>.META.</code>.
* <p>
* This class is "read-only" in that the locations of the catalog tables cannot
* be explicitly set. Instead, ZooKeeper is used to learn of the availability
* and location of ROOT. ROOT is used to learn of the location of META. If not
* available in ROOT, ZooKeeper is used to monitor for a new location of META.
*/
public class CatalogTracker {
private static final Log LOG = LogFactory.getLog(CatalogTracker.class);
private final ServerConnection connection;
private final ZooKeeperWatcher zookeeper;
private final RootRegionTracker rootRegionTracker;
private final MetaNodeTracker metaNodeTracker;
private final AtomicBoolean metaAvailable = new AtomicBoolean(false);
private HServerAddress metaLocation;
private final int defaultTimeout;
public static final byte [] ROOT_REGION =
HRegionInfo.ROOT_REGIONINFO.getRegionName();
public static final byte [] META_REGION =
HRegionInfo.FIRST_META_REGIONINFO.getRegionName();
/**
* Constructs the catalog tracker. Find current state of catalog tables and
* begin active tracking by executing {@link #start()}.
* @param zk
* @param connection server connection
* @param abortable if fatal exception
* @throws IOException
*/
public CatalogTracker(final ZooKeeperWatcher zk,
final ServerConnection connection, final Abortable abortable,
final int defaultTimeout)
throws IOException {
this.zookeeper = zk;
this.connection = connection;
this.rootRegionTracker = new RootRegionTracker(zookeeper, abortable);
this.metaNodeTracker = new MetaNodeTracker(zookeeper, this);
this.defaultTimeout = defaultTimeout;
}
/**
* Starts the catalog tracker.
* <p>
* Determines current availability of catalog tables and ensures all further
* transitions of either region is tracked.
* @throws IOException
* @throws InterruptedException
*/
public void start() throws IOException, InterruptedException {
// Register listeners with zk
zookeeper.registerListener(rootRegionTracker);
zookeeper.registerListener(metaNodeTracker);
// Start root tracking
rootRegionTracker.start();
// Determine meta assignment; may not work because root and meta not yet
// deployed.
getMetaServerConnection(true);
}
/**
* Gets the current location for <code>-ROOT-</code> or null if location is
* not currently available.
* @return location of root, null if not available
* @throws InterruptedException
*/
public HServerAddress getRootLocation() throws InterruptedException {
return this.rootRegionTracker.getRootRegionLocation();
}
/**
* @return Location of meta or null if not yet available.
*/
public HServerAddress getMetaLocation() {
return this.metaLocation;
}
/**
* Waits indefinitely for availability of <code>-ROOT-</code>. Used during
* cluster startup.
* @throws InterruptedException if interrupted while waiting
*/
public void waitForRoot()
throws InterruptedException {
rootRegionTracker.getRootRegionLocation();
}
/**
* Gets the current location for <code>-ROOT-</code> if available and waits
* for up to the specified timeout if not immediately available. Returns null
* if the timeout elapses before root is available.
* @param timeout maximum time to wait for root availability, in milliseconds
* @return location of root
* @throws InterruptedException if interrupted while waiting
* @throws NotAllMetaRegionsOnlineException if root not available before
* timeout
*/
public HServerAddress waitForRoot(final long timeout)
throws InterruptedException, NotAllMetaRegionsOnlineException {
HServerAddress address = rootRegionTracker.waitRootRegionLocation(timeout);
if (address == null) {
throw new NotAllMetaRegionsOnlineException("Timed out; " + timeout + "ms");
}
return address;
}
/**
* Gets a connection to the server hosting root, as reported by ZooKeeper,
* waiting up to the specified timeout for availability.
* @see #waitForRoot(long) for additional information
* @return connection to server hosting root
* @throws InterruptedException
* @throws NotAllMetaRegionsOnlineException if timed out waiting
* @throws IOException
*/
public HRegionInterface waitForRootServerConnection(long timeout)
throws InterruptedException, NotAllMetaRegionsOnlineException, IOException {
return getCachedConnection(waitForRoot(timeout));
}
/**
* Gets a connection to the server hosting root, as reported by ZooKeeper,
* waiting for the default timeout specified on instantiation.
* @see #waitForRoot(long) for additional information
* @return connection to server hosting root
* @throws NotAllMetaRegionsOnlineException if timed out waiting
* @throws IOException
*/
public HRegionInterface waitForRootServerConnectionDefault()
throws NotAllMetaRegionsOnlineException, IOException {
try {
return getCachedConnection(waitForRoot(defaultTimeout));
} catch (InterruptedException e) {
throw new NotAllMetaRegionsOnlineException("Interrupted");
}
}
/**
* Gets a connection to the server hosting root, as reported by ZooKeeper,
* if available. Returns null if no location is immediately available.
* @return connection to server hosting root, null if not available
* @throws IOException
* @throws InterruptedException
*/
private HRegionInterface getRootServerConnection()
throws IOException, InterruptedException {
HServerAddress address = rootRegionTracker.getRootRegionLocation();
if (address == null) {
return null;
}
return getCachedConnection(address);
}
/**
* Gets a connection to the server currently hosting <code>.META.</code> or
* null if location is not currently available.
* <p>
* If a location is known, a connection to the cached location is returned.
* If refresh is true, the cached connection is verified first before
* returning. If the connection is not valid, it is reset and rechecked.
* <p>
* If no location for meta is currently known, method checks ROOT for a new
* location, verifies META is currently there, and returns a cached connection
* to the server hosting META.
*
* @return connection to server hosting meta, null if location not available
* @throws IOException
* @throws InterruptedException
*/
private HRegionInterface getMetaServerConnection(boolean refresh)
throws IOException, InterruptedException {
synchronized(metaAvailable) {
if(metaAvailable.get()) {
HRegionInterface current = getCachedConnection(metaLocation);
if(!refresh) {
return current;
}
if(verifyRegionLocation(current, META_REGION)) {
return current;
}
resetMetaLocation();
}
HRegionInterface rootConnection = getRootServerConnection();
if(rootConnection == null) {
return null;
}
HServerAddress newLocation = MetaReader.readMetaLocation(rootConnection);
if(newLocation == null) {
return null;
}
HRegionInterface newConnection = getCachedConnection(newLocation);
if(verifyRegionLocation(newConnection, META_REGION)) {
setMetaLocation(newLocation);
return newConnection;
}
return null;
}
}
/**
* Waits indefinitely for availability of <code>.META.</code>. Used during
* cluster startup.
* @throws InterruptedException if interrupted while waiting
*/
public void waitForMeta() throws InterruptedException {
synchronized(metaAvailable) {
while(!metaAvailable.get()) {
metaAvailable.wait();
}
}
}
/**
* Gets the current location for <code>.META.</code> if available and waits
* for up to the specified timeout if not immediately available. Throws an
* exception if timed out waiting.
* @param timeout maximum time to wait for meta availability, in milliseconds
* @return location of meta
* @throws InterruptedException if interrupted while waiting
* @throws IOException unexpected exception connecting to meta server
* @throws NotAllMetaRegionsOnlineException if meta not available before
* timeout
*/
public HServerAddress waitForMeta(long timeout)
throws InterruptedException, IOException, NotAllMetaRegionsOnlineException {
long stop = System.currentTimeMillis() + timeout;
synchronized(metaAvailable) {
if(getMetaServerConnection(true) != null) {
return metaLocation;
}
while(!metaAvailable.get() &&
(timeout == 0 || System.currentTimeMillis() < stop)) {
metaAvailable.wait(timeout);
}
if(getMetaServerConnection(true) == null) {
throw new NotAllMetaRegionsOnlineException(
"Timed out (" + timeout + "ms");
}
return metaLocation;
}
}
/**
* Gets a connection to the server hosting meta, as reported by ZooKeeper,
* waiting up to the specified timeout for availability.
* @see #waitForMeta(long) for additional information
* @return connection to server hosting meta
* @throws InterruptedException
* @throws NotAllMetaRegionsOnlineException if timed out waiting
* @throws IOException
*/
public HRegionInterface waitForMetaServerConnection(long timeout)
throws InterruptedException, NotAllMetaRegionsOnlineException, IOException {
return getCachedConnection(waitForMeta(timeout));
}
/**
* Gets a connection to the server hosting meta, as reported by ZooKeeper,
* waiting up to the specified timeout for availability.
* @see #waitForMeta(long) for additional information
* @return connection to server hosting meta
* @throws NotAllMetaRegionsOnlineException if timed out or interrupted
* @throws IOException
*/
public HRegionInterface waitForMetaServerConnectionDefault()
throws NotAllMetaRegionsOnlineException, IOException {
try {
return getCachedConnection(waitForMeta(defaultTimeout));
} catch (InterruptedException e) {
throw new NotAllMetaRegionsOnlineException("Interrupted");
}
}
private void resetMetaLocation() {
LOG.info("Current cached META location is not valid, resetting");
metaAvailable.set(false);
metaLocation = null;
}
private void setMetaLocation(HServerAddress metaLocation) {
LOG.info("Found new META location, " + metaLocation);
metaAvailable.set(true);
this.metaLocation = metaLocation;
// no synchronization because these are private and already under lock
metaAvailable.notifyAll();
}
private HRegionInterface getCachedConnection(HServerAddress address)
throws IOException {
HRegionInterface protocol = null;
try {
protocol = connection.getHRegionConnection(address, false);
} catch (RetriesExhaustedException e) {
if (e.getCause() != null && e.getCause() instanceof ConnectException) {
// Catch this; presume it means the cached connection has gone bad.
} else {
throw e;
}
}
return protocol;
}
private boolean verifyRegionLocation(HRegionInterface metaServer,
byte [] regionName) {
try {
return metaServer.getRegionInfo(regionName) != null;
} catch (NotServingRegionException e) {
return false;
}
}
/**
* Check if <code>hsi</code> was carrying <code>-ROOT-</code> or
* <code>.META.</code> and if so, clear out old locations.
* @param hsi Server that has crashed/shutdown.
* @throws InterruptedException
* @throws KeeperException
* @return Pair of booleans; if this server was carrying root, then first
* boolean is set, if server was carrying meta, then second boolean set.
*/
public Pair<Boolean, Boolean> processServerShutdown(final HServerInfo hsi)
throws InterruptedException, KeeperException {
Pair<Boolean, Boolean> result = new Pair<Boolean, Boolean>(false, false);
HServerAddress rootHsa = getRootLocation();
if (rootHsa == null) {
LOG.info("-ROOT- is not assigned; continuing");
} else if (hsi.getServerAddress().equals(rootHsa)) {
result.setFirst(true);
LOG.info(hsi.getServerName() + " carrying -ROOT-; deleting " +
"-ROOT- location from meta");
RootLocationEditor.deleteRootLocation(this.zookeeper);
}
HServerAddress metaHsa = getMetaLocation();
if (metaHsa == null) {
LOG.info(".META. is not assigned; continuing");
} else if (hsi.getServerAddress().equals(metaHsa)) {
LOG.info(hsi.getServerName() + " carrying .META.; unsetting " +
".META. location");
result.setSecond(true);
resetMetaLocation();
}
return result;
}
}

215
src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java Normal file
View File

@ -0,0 +1,215 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.catalog;
import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Writables;
/**
* Writes region and assignment information to <code>.META.</code>.
* <p>
* Uses the {@link CatalogTracker} to obtain locations and connections to
* catalogs.
*/
public class MetaEditor {
private static final Log LOG = LogFactory.getLog(MetaEditor.class);
/**
* Adds a META row for the specified new region.
* @param info region information
* @throws IOException if problem connecting or updating meta
*/
public static void addRegionToMeta(CatalogTracker catalogTracker,
HRegionInfo regionInfo)
throws IOException {
Put put = new Put(regionInfo.getRegionName());
put.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER,
Writables.getBytes(regionInfo));
catalogTracker.waitForMetaServerConnectionDefault().put(
CatalogTracker.META_REGION, put);
LOG.info("Added region " + regionInfo.getRegionNameAsString() + " to META");
}
/**
* Offline parent in meta.
* Used when splitting.
* @param catalogTracker
* @param parent
* @param a Split daughter region A
* @param b Split daughter region B
* @throws NotAllMetaRegionsOnlineException
* @throws IOException
*/
public static void offlineParentInMeta(CatalogTracker catalogTracker,
HRegionInfo parent, final HRegionInfo a, final HRegionInfo b)
throws NotAllMetaRegionsOnlineException, IOException {
HRegionInfo copyOfParent = new HRegionInfo(parent);
copyOfParent.setOffline(true);
copyOfParent.setSplit(true);
Put put = new Put(copyOfParent.getRegionName());
addRegionInfo(put, copyOfParent);
put.add(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER,
HConstants.EMPTY_BYTE_ARRAY);
put.add(HConstants.CATALOG_FAMILY, HConstants.STARTCODE_QUALIFIER,
HConstants.EMPTY_BYTE_ARRAY);
put.add(HConstants.CATALOG_FAMILY, HConstants.SPLITA_QUALIFIER,
Writables.getBytes(a));
put.add(HConstants.CATALOG_FAMILY, HConstants.SPLITB_QUALIFIER,
Writables.getBytes(b));
catalogTracker.waitForMetaServerConnectionDefault().put(CatalogTracker.META_REGION, put);
LOG.info("Offlined parent region " + parent.getRegionNameAsString() +
" in META");
}
public static void addDaughter(final CatalogTracker catalogTracker,
final HRegionInfo regionInfo, final HServerInfo serverInfo)
throws NotAllMetaRegionsOnlineException, IOException {
HRegionInterface server = catalogTracker.waitForMetaServerConnectionDefault();
byte [] catalogRegionName = CatalogTracker.META_REGION;
Put put = new Put(regionInfo.getRegionName());
addRegionInfo(put, regionInfo);
if (serverInfo != null) addLocation(put, serverInfo);
server.put(catalogRegionName, put);
LOG.info("Added daughter " + regionInfo.getRegionNameAsString() +
" in region " + Bytes.toString(catalogRegionName) + " with " +
"server=" + serverInfo.getHostnamePort() + ", " +
"startcode=" + serverInfo.getStartCode());
}
/**
* Updates the location of the specified META region in ROOT to be the
* specified server hostname and startcode.
* <p>
* Uses passed catalog tracker to get a connection to the server hosting
* ROOT and makes edits to that region.
*
* @param catalogTracker catalog tracker
* @param regionInfo region to update location of
* @param serverInfo server the region is located on
* @throws IOException
*/
public static void updateMetaLocation(CatalogTracker catalogTracker,
HRegionInfo regionInfo, HServerInfo serverInfo)
throws IOException {
HRegionInterface server = catalogTracker.waitForRootServerConnectionDefault();
updateLocation(server, CatalogTracker.ROOT_REGION, regionInfo, serverInfo);
}
/**
* Updates the location of the specified region in META to be the specified
* server hostname and startcode.
* <p>
* Uses passed catalog tracker to get a connection to the server hosting
* META and makes edits to that region.
*
* @param catalogTracker catalog tracker
* @param regionInfo region to update location of
* @param serverInfo server the region is located on
* @throws IOException
*/
public static void updateRegionLocation(CatalogTracker catalogTracker,
HRegionInfo regionInfo, HServerInfo serverInfo)
throws IOException {
updateLocation(catalogTracker.waitForMetaServerConnectionDefault(),
CatalogTracker.META_REGION, regionInfo, serverInfo);
}
/**
* Updates the location of the specified region to be the specified server.
* <p>
* Connects to the specified server which should be hosting the specified
* catalog region name to perform the edit.
*
* @param server connection to server hosting catalog region
* @param catalogRegionName name of catalog region being updated
* @param regionInfo region to update location of
* @param serverInfo server the region is located on
* @throws IOException
*/
public static void updateLocation(HRegionInterface server,
byte [] catalogRegionName, HRegionInfo regionInfo, HServerInfo serverInfo)
throws IOException {
Put put = new Put(regionInfo.getRegionName());
addLocation(put, serverInfo);
server.put(catalogRegionName, put);
LOG.info("Updated row " + regionInfo.getRegionNameAsString() +
" in region " + Bytes.toString(catalogRegionName) + " with " +
"server=" + serverInfo.getHostnamePort() + ", " +
"startcode=" + serverInfo.getStartCode());
}
/**
* Deletes the specified region from META.
* @param catalogTracker
* @param regionInfo region to be deleted from META
* @throws IOException
*/
public static void deleteRegion(CatalogTracker catalogTracker,
HRegionInfo regionInfo)
throws IOException {
Delete delete = new Delete(regionInfo.getRegionName());
catalogTracker.waitForMetaServerConnectionDefault().delete(
CatalogTracker.META_REGION, delete);
LOG.info("Deleted region " + regionInfo.getRegionNameAsString() + " from META");
}
/**
* Updates the region information for the specified region in META.
* @param catalogTracker
* @param regionInfo region to be updated in META
* @throws IOException
*/
public static void updateRegionInfo(CatalogTracker catalogTracker,
HRegionInfo regionInfo)
throws IOException {
Put put = new Put(regionInfo.getRegionName());
addRegionInfo(put, regionInfo);
catalogTracker.waitForMetaServerConnectionDefault().put(
CatalogTracker.META_REGION, put);
LOG.info("Updated region " + regionInfo.getRegionNameAsString() + " in META");
}
private static Put addRegionInfo(final Put p, final HRegionInfo hri)
throws IOException {
p.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER,
Writables.getBytes(hri));
return p;
}
private static Put addLocation(final Put p, final HServerInfo hsi) {
p.add(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER,
Bytes.toBytes(hsi.getHostnamePort()));
p.add(HConstants.CATALOG_FAMILY, HConstants.STARTCODE_QUALIFIER,
Bytes.toBytes(hsi.getStartCode()));
return p;
}
}

309
src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java Normal file
View File

@ -0,0 +1,309 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.catalog;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.NavigableMap;
import java.util.NavigableSet;
import java.util.TreeMap;
import java.util.TreeSet;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.util.Writables;
/**
* Reads region and assignment information from <code>.META.</code>.
* <p>
* Uses the {@link CatalogTracker} to obtain locations and connections to
* catalogs.
*/
public class MetaReader {
/**
* Performs a full scan of <code>.META.</code>.
* <p>
* Returns a map of every region to it's currently assigned server, according
* to META. If the region does not have an assignment it will have a null
* value in the map.
*
* @return map of regions to their currently assigned server
* @throws IOException
*/
public static Map<HRegionInfo,HServerAddress> fullScan(
CatalogTracker catalogTracker)
throws IOException {
HRegionInterface metaServer =
catalogTracker.waitForMetaServerConnectionDefault();
Map<HRegionInfo,HServerAddress> allRegions =
new TreeMap<HRegionInfo,HServerAddress>();
Scan scan = new Scan();
scan.addFamily(HConstants.CATALOG_FAMILY);
long scannerid = metaServer.openScanner(
HRegionInfo.FIRST_META_REGIONINFO.getRegionName(), scan);
try {
Result data;
while((data = metaServer.next(scannerid)) != null) {
if (!data.isEmpty()) {
Pair<HRegionInfo,HServerAddress> region =
metaRowToRegionPair(data);
allRegions.put(region.getFirst(), region.getSecond());
}
}
} finally {
metaServer.close(scannerid);
}
return allRegions;
}
/**
* Reads the location of META from ROOT.
* @param metaServer connection to server hosting ROOT
* @return location of META in ROOT, null if not available
* @throws IOException
*/
public static HServerAddress readMetaLocation(HRegionInterface metaServer)
throws IOException {
return readLocation(metaServer, CatalogTracker.ROOT_REGION,
CatalogTracker.META_REGION);
}
/**
* Reads the location of the specified region from META.
* @param catalogTracker
* @param regionName region to read location of
* @return location of region in META, null if not available
* @throws IOException
*/
public static HServerAddress readRegionLocation(CatalogTracker catalogTracker,
byte [] regionName)
throws IOException {
return readLocation(catalogTracker.waitForMetaServerConnectionDefault(),
CatalogTracker.META_REGION, regionName);
}
private static HServerAddress readLocation(HRegionInterface metaServer,
byte [] catalogRegionName, byte [] regionName)
throws IOException {
Result r = null;
try {
r = metaServer.get(catalogRegionName,
new Get(regionName).addColumn(HConstants.CATALOG_FAMILY,
HConstants.SERVER_QUALIFIER));
} catch (java.net.ConnectException e) {
if (e.getMessage() != null &&
e.getMessage().contains("Connection refused")) {
// Treat this exception + message as unavailable catalog table. Catch it
// and fall through to return a null
} else {
throw e;
}
} catch (IOException e) {
if (e.getCause() != null && e.getCause() instanceof IOException &&
e.getCause().getMessage() != null &&
e.getCause().getMessage().contains("Connection reset by peer")) {
// Treat this exception + message as unavailable catalog table. Catch it
// and fall through to return a null
} else {
throw e;
}
}
if (r == null || r.isEmpty()) {
return null;
}
byte [] value = r.getValue(HConstants.CATALOG_FAMILY,
HConstants.SERVER_QUALIFIER);
return new HServerAddress(Bytes.toString(value));
}
/**
* Gets the region info and assignment for the specified region from META.
* @param catalogTracker
* @param regionName
* @return region info and assignment from META, null if not available
* @throws IOException
*/
public static Pair<HRegionInfo, HServerAddress> getRegion(
CatalogTracker catalogTracker, byte [] regionName)
throws IOException {
Get get = new Get(regionName);
get.addFamily(HConstants.CATALOG_FAMILY);
Result r = catalogTracker.waitForMetaServerConnectionDefault().get(
CatalogTracker.META_REGION, get);
if(r == null || r.isEmpty()) {
return null;
}
return metaRowToRegionPair(r);
}
public static Pair<HRegionInfo, HServerAddress> metaRowToRegionPair(
Result data) throws IOException {
HRegionInfo info = Writables.getHRegionInfo(
data.getValue(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER));
final byte[] value = data.getValue(HConstants.CATALOG_FAMILY,
HConstants.SERVER_QUALIFIER);
if (value != null && value.length > 0) {
HServerAddress server = new HServerAddress(Bytes.toString(value));
return new Pair<HRegionInfo,HServerAddress>(info, server);
} else {
return new Pair<HRegionInfo, HServerAddress>(info, null);
}
}
/**
* Checks if the specified table exists. Looks at the META table hosted on
* the specified server.
* @param metaServer server hosting meta
* @param tableName table to check
* @return true if the table exists in meta, false if not
* @throws IOException
*/
public static boolean tableExists(CatalogTracker catalogTracker,
String tableName)
throws IOException {
HRegionInterface metaServer =
catalogTracker.waitForMetaServerConnectionDefault();
byte[] firstRowInTable = Bytes.toBytes(tableName + ",,");
Scan scan = new Scan(firstRowInTable);
scan.addColumn(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER);
long scannerid = metaServer.openScanner(
HRegionInfo.FIRST_META_REGIONINFO.getRegionName(), scan);
try {
Result data = metaServer.next(scannerid);
if (data != null && data.size() > 0) {
HRegionInfo info = Writables.getHRegionInfo(
data.getValue(HConstants.CATALOG_FAMILY,
HConstants.REGIONINFO_QUALIFIER));
if (info.getTableDesc().getNameAsString().equals(tableName)) {
// A region for this table already exists. Ergo table exists.
return true;
}
}
return false;
} finally {
metaServer.close(scannerid);
}
}
/**
* Gets all of the regions of the specified table from META.
* @param catalogTracker
* @param tableName
* @return
* @throws IOException
*/
public static List<HRegionInfo> getTableRegions(CatalogTracker catalogTracker,
byte [] tableName)
throws IOException {
HRegionInterface metaServer =
catalogTracker.waitForMetaServerConnectionDefault();
List<HRegionInfo> regions = new ArrayList<HRegionInfo>();
String tableString = Bytes.toString(tableName);
byte[] firstRowInTable = Bytes.toBytes(tableString + ",,");
Scan scan = new Scan(firstRowInTable);
scan.addColumn(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER);
long scannerid = metaServer.openScanner(
HRegionInfo.FIRST_META_REGIONINFO.getRegionName(), scan);
try {
Result data;
while((data = metaServer.next(scannerid)) != null) {
if (data != null && data.size() > 0) {
HRegionInfo info = Writables.getHRegionInfo(
data.getValue(HConstants.CATALOG_FAMILY,
HConstants.REGIONINFO_QUALIFIER));
if (info.getTableDesc().getNameAsString().equals(tableString)) {
regions.add(info);
} else {
break;
}
}
}
return regions;
} finally {
metaServer.close(scannerid);
}
}
public static List<Pair<HRegionInfo, HServerAddress>>
getTableRegionsAndLocations(CatalogTracker catalogTracker, String tableName)
throws IOException {
HRegionInterface metaServer =
catalogTracker.waitForMetaServerConnectionDefault();
List<Pair<HRegionInfo, HServerAddress>> regions =
new ArrayList<Pair<HRegionInfo, HServerAddress>>();
byte[] firstRowInTable = Bytes.toBytes(tableName + ",,");
Scan scan = new Scan(firstRowInTable);
scan.addFamily(HConstants.CATALOG_FAMILY);
long scannerid = metaServer.openScanner(
HRegionInfo.FIRST_META_REGIONINFO.getRegionName(), scan);
try {
Result data;
while((data = metaServer.next(scannerid)) != null) {
if (data != null && data.size() > 0) {
Pair<HRegionInfo, HServerAddress> region = metaRowToRegionPair(data);
if (region.getFirst().getTableDesc().getNameAsString().equals(
tableName)) {
regions.add(region);
} else {
break;
}
}
}
return regions;
} finally {
metaServer.close(scannerid);
}
}
public static NavigableMap<HRegionInfo, Result>
getServerRegions(CatalogTracker catalogTracker, final HServerInfo hsi)
throws IOException {
HRegionInterface metaServer =
catalogTracker.waitForMetaServerConnectionDefault();
NavigableMap<HRegionInfo, Result> hris = new TreeMap<HRegionInfo, Result>();
Scan scan = new Scan();
scan.addFamily(HConstants.CATALOG_FAMILY);
long scannerid = metaServer.openScanner(
HRegionInfo.FIRST_META_REGIONINFO.getRegionName(), scan);
try {
Result result;
while((result = metaServer.next(scannerid)) != null) {
if (result != null && result.size() > 0) {
Pair<HRegionInfo, HServerAddress> pair = metaRowToRegionPair(result);
if (!pair.getSecond().equals(hsi.getServerAddress())) continue;
hris.put(pair.getFirst(), result);
}
}
return hris;
} finally {
metaServer.close(scannerid);
}
}
}

72
src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java Normal file
View File

@ -0,0 +1,72 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.catalog;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.zookeeper.ZKUtil;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher;
import org.apache.zookeeper.KeeperException;
/**
* Makes changes to the location of <code>-ROOT-</code> in ZooKeeper.
*/
public class RootLocationEditor {
private static final Log LOG = LogFactory.getLog(RootLocationEditor.class);
/**
* Deletes the location of <code>-ROOT-</code> in ZooKeeper.
* @param zookeeper zookeeper reference
* @throws KeeperException unexpected zookeeper exception
*/
public static void deleteRootLocation(ZooKeeperWatcher zookeeper)
throws KeeperException {
LOG.info("Unsetting ROOT region location in ZooKeeper");
try {
// Just delete the node. Don't need any watches, only we will create it.
ZKUtil.deleteNode(zookeeper, zookeeper.rootServerZNode);
} catch(KeeperException.NoNodeException nne) {
// Has already been deleted
}
}
/**
* Sets the location of <code>-ROOT-</code> in ZooKeeper to the
* specified server address.
* @param zookeeper zookeeper reference
* @param location server address hosting root
* @throws KeeperException unexpected zookeeper exception
*/
public static void setRootLocation(ZooKeeperWatcher zookeeper,
HServerAddress location)
throws KeeperException {
LOG.info("Setting ROOT region location in ZooKeeper as " + location);
try {
ZKUtil.createAndWatch(zookeeper, zookeeper.rootServerZNode,
Bytes.toBytes(location.toString()));
} catch(KeeperException.NodeExistsException nee) {
LOG.debug("ROOT region location already existed, updated location");
ZKUtil.setData(zookeeper, zookeeper.rootServerZNode,
Bytes.toBytes(location.toString()));
}
}
}

7
src/main/java/org/apache/hadoop/hbase/client/Get.java
View File

@ -60,7 +60,7 @@ import java.util.TreeSet;
* <p>
* To add a filter, execute {@link #setFilter(Filter) setFilter}.
*/
public class Get implements Writable, Row, Comparable<Row> {
public class Get implements Writable {
private static final byte GET_VERSION = (byte)1;
private byte [] row = null;
@ -325,11 +325,6 @@ public class Get implements Writable, Row, Comparable<Row> {
return sb.toString();
}
//Row
public int compareTo(Row other) {
return Bytes.compareTo(this.getRow(), other.getRow());
}
//Writable
public void readFields(final DataInput in)
throws IOException {

477
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
View File

@ -19,35 +19,39 @@
*/
package org.apache.hadoop.hbase.client;
import java.io.IOException;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.NavigableMap;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Abortable;
import org.apache.hadoop.hbase.ClusterStatus;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HRegionLocation;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.RegionException;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.TableExistsException;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.UnknownRegionException;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.catalog.CatalogTracker;
import org.apache.hadoop.hbase.catalog.MetaReader;
import org.apache.hadoop.hbase.ipc.HMasterInterface;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.MetaUtils;
import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.util.Writables;
import org.apache.hadoop.io.BooleanWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.ipc.RemoteException;
import java.io.IOException;
import java.util.Arrays;
import java.util.Map;
import java.util.NavigableMap;
/**
* Provides an interface to manage HBase database table metadata + general
* administrative functions. Use HBaseAdmin to create, drop, list, enable and
@ -55,27 +59,56 @@ import java.util.NavigableMap;
*
* See {@link HTable} to add, update, and delete data from an individual table.
*/
public class HBaseAdmin {
public class HBaseAdmin implements Abortable {
private final Log LOG = LogFactory.getLog(this.getClass().getName());
// private final HConnection connection;
final HConnection connection;
private volatile Configuration conf;
private final long pause;
private final int numRetries;
private volatile HMasterInterface master;
/**
* Lazily instantiated. Use {@link #getCatalogTracker()} to ensure you get
* an instance rather than a null.
*/
private CatalogTracker catalogTracker = null;
/**
* Constructor
*
* @param conf Configuration object
* @throws MasterNotRunningException if the master is not running
* @throws ZooKeeperConnectionException if unable to connect to zookeeper
*/
public HBaseAdmin(Configuration conf) throws MasterNotRunningException {
public HBaseAdmin(Configuration conf)
throws MasterNotRunningException, ZooKeeperConnectionException {
this.connection = HConnectionManager.getConnection(conf);
this.conf = conf;
this.pause = conf.getLong("hbase.client.pause", 30 * 1000);
this.numRetries = conf.getInt("hbase.client.retries.number", 5);
this.master = connection.getMaster();
this.connection.getMaster();
}
private synchronized CatalogTracker getCatalogTracker()
throws ZooKeeperConnectionException, IOException {
if (this.catalogTracker == null) {
this.catalogTracker = new CatalogTracker(this.connection.getZooKeeperWatcher(),
ServerConnectionManager.getConnection(conf), this,
this.conf.getInt("hbase.admin.catalog.timeout", 10 * 1000));
try {
this.catalogTracker.start();
} catch (InterruptedException e) {
// Let it out as an IOE for now until we redo all so tolerate IEs
Thread.currentThread().interrupt();
throw new IOException("Interrupted", e);
}
}
return this.catalogTracker;
}
@Override
public void abort(String why, Throwable e) {
// Currently does nothing but throw the passed message and exception
throw new RuntimeException(why, e);
}
/** @return HConnection used by this object. */
@ -84,15 +117,21 @@ public class HBaseAdmin {
}
/**
* Get a connection to the currently set master.
* @return proxy connection to master server for this instance
* @throws MasterNotRunningException if the master is not running
* @throws ZooKeeperConnectionException if unable to connect to zookeeper
*/
public HMasterInterface getMaster() throws MasterNotRunningException{
public HMasterInterface getMaster()
throws MasterNotRunningException, ZooKeeperConnectionException {
return this.connection.getMaster();
}
/** @return - true if the master server is running */
public boolean isMasterRunning() {
/** @return - true if the master server is running
* @throws ZooKeeperConnectionException
* @throws MasterNotRunningException */
public boolean isMasterRunning()
throws MasterNotRunningException, ZooKeeperConnectionException {
return this.connection.isMasterRunning();
}
@ -100,9 +139,10 @@ public class HBaseAdmin {
* @param tableName Table to check.
* @return True if table exists already.
* @throws MasterNotRunningException if the master is not running
* @throws ZooKeeperConnectionException if unable to connect to zookeeper
*/
public boolean tableExists(final String tableName)
throws MasterNotRunningException {
throws MasterNotRunningException, ZooKeeperConnectionException {
return tableExists(Bytes.toBytes(tableName));
}
@ -110,12 +150,11 @@ public class HBaseAdmin {
* @param tableName Table to check.
* @return True if table exists already.
* @throws MasterNotRunningException if the master is not running
* @throws ZooKeeperConnectionException if unable to connect to zookeeper
*/
public boolean tableExists(final byte [] tableName)
throws MasterNotRunningException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
throws MasterNotRunningException, ZooKeeperConnectionException {
connection.isMasterRunning();
return connection.tableExists(tableName);
}
@ -147,8 +186,9 @@ public class HBaseAdmin {
private long getPauseTime(int tries) {
int triesCount = tries;
if (triesCount >= HConstants.RETRY_BACKOFF.length)
if (triesCount >= HConstants.RETRY_BACKOFF.length) {
triesCount = HConstants.RETRY_BACKOFF.length - 1;
}
return this.pause * HConstants.RETRY_BACKOFF[triesCount];
}
@ -257,7 +297,7 @@ public class HBaseAdmin {
try {
Thread.sleep(getPauseTime(tries));
} catch (InterruptedException e) {
// continue
// Just continue; ignore the interruption.
}
}
}
@ -277,12 +317,9 @@ public class HBaseAdmin {
*/
public void createTableAsync(HTableDescriptor desc, byte [][] splitKeys)
throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
HTableDescriptor.isLegalTableName(desc.getName());
try {
this.master.createTable(desc, splitKeys);
getMaster().createTable(desc, splitKeys);
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
}
@ -307,13 +344,11 @@ public class HBaseAdmin {
* @throws IOException if a remote or network exception occurs
*/
public void deleteTable(final byte [] tableName) throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
isMasterRunning();
HTableDescriptor.isLegalTableName(tableName);
HRegionLocation firstMetaServer = getFirstMetaServerForTable(tableName);
try {
this.master.deleteTable(tableName);
getMaster().deleteTable(tableName);
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
}
@ -401,21 +436,20 @@ public class HBaseAdmin {
* @throws IOException if a remote or network exception occurs
*/
public void enableTable(final byte [] tableName) throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
isMasterRunning();
// Wait until all regions are enabled
boolean enabled = false;
for (int tries = 0; tries < this.numRetries; tries++) {
try {
this.master.enableTable(tableName);
getMaster().enableTable(tableName);
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
}
enabled = isTableEnabled(tableName);
if (enabled) break;
if (enabled) {
break;
}
long sleep = getPauseTime(tries);
if (LOG.isDebugEnabled()) {
LOG.debug("Sleeping= " + sleep + "ms, waiting for all regions to be " +
@ -431,9 +465,10 @@ public class HBaseAdmin {
Bytes.toString(tableName));
}
}
if (!enabled)
if (!enabled) {
throw new IOException("Unable to enable table " +
Bytes.toString(tableName));
}
LOG.info("Enabled table " + Bytes.toString(tableName));
}
@ -458,20 +493,20 @@ public class HBaseAdmin {
* @throws IOException if a remote or network exception occurs
*/
public void disableTable(final byte [] tableName) throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
isMasterRunning();
// Wait until all regions are disabled
boolean disabled = false;
for (int tries = 0; tries < this.numRetries; tries++) {
try {
this.master.disableTable(tableName);
getMaster().disableTable(tableName);
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
}
disabled = isTableDisabled(tableName);
if (disabled) break;
if (disabled) {
break;
}
if (LOG.isDebugEnabled()) {
LOG.debug("Sleep. Waiting for all regions to be disabled from " +
Bytes.toString(tableName));
@ -560,12 +595,9 @@ public class HBaseAdmin {
*/
public void addColumn(final byte [] tableName, HColumnDescriptor column)
throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
HTableDescriptor.isLegalTableName(tableName);
try {
this.master.addColumn(tableName, column);
getMaster().addColumn(tableName, column);
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
}
@ -594,12 +626,8 @@ public class HBaseAdmin {
*/
public void deleteColumn(final byte [] tableName, final byte [] columnName)
throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
HTableDescriptor.isLegalTableName(tableName);
try {
this.master.deleteColumn(tableName, columnName);
getMaster().deleteColumn(tableName, columnName);
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
}
@ -613,12 +641,25 @@ public class HBaseAdmin {
* @param columnName name of column to be modified
* @param descriptor new column descriptor to use
* @throws IOException if a remote or network exception occurs
* @deprecated The <code>columnName</code> is redundant. Use {@link #addColumn(String, HColumnDescriptor)}
*/
public void modifyColumn(final String tableName, final String columnName,
HColumnDescriptor descriptor)
throws IOException {
modifyColumn(Bytes.toBytes(tableName), Bytes.toBytes(columnName),
descriptor);
modifyColumn(tableName, descriptor);
}
/**
* Modify an existing column family on a table.
* Asynchronous operation.
*
* @param tableName name of table
* @param descriptor new column descriptor to use
* @throws IOException if a remote or network exception occurs
*/
public void modifyColumn(final String tableName, HColumnDescriptor descriptor)
throws IOException {
modifyColumn(Bytes.toBytes(tableName), descriptor);
}
/**
@ -629,56 +670,71 @@ public class HBaseAdmin {
* @param columnName name of column to be modified
* @param descriptor new column descriptor to use
* @throws IOException if a remote or network exception occurs
* @deprecated The <code>columnName</code> is redundant. Use {@link #modifyColumn(byte[], HColumnDescriptor)}
*/
public void modifyColumn(final byte [] tableName, final byte [] columnName,
HColumnDescriptor descriptor)
throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
HTableDescriptor.isLegalTableName(tableName);
modifyColumn(tableName, descriptor);
}
/**
* Modify an existing column family on a table.
* Asynchronous operation.
*
* @param tableName name of table
* @param descriptor new column descriptor to use
* @throws IOException if a remote or network exception occurs
*/
public void modifyColumn(final byte [] tableName, HColumnDescriptor descriptor)
throws IOException {
try {
this.master.modifyColumn(tableName, columnName, descriptor);
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
getMaster().modifyColumn(tableName, descriptor);
} catch (RemoteException re) {
// Convert RE exceptions in here; client shouldn't have to deal with them,
// at least w/ the type of exceptions that come out of this method:
// TableNotFoundException, etc.
throw RemoteExceptionHandler.decodeRemoteException(re);
}
}
/**
* Close a region. For expert-admins.
* Asynchronous operation.
*
* @param regionname region name to close
* @param args Optional server name. Otherwise, we'll send close to the
* server registered in .META.
* @param hostAndPort If supplied, we'll use this location rather than
* the one currently in <code>.META.</code>
* @throws IOException if a remote or network exception occurs
*/
public void closeRegion(final String regionname, final Object... args)
public void closeRegion(final String regionname, final String hostAndPort)
throws IOException {
closeRegion(Bytes.toBytes(regionname), args);
closeRegion(Bytes.toBytes(regionname), hostAndPort);
}
/**
* Close a region. For expert-admins.
* Asynchronous operation.
*
* @param regionname region name to close
* @param args Optional server name. Otherwise, we'll send close to the
* server registered in .META.
* @param hostAndPort If supplied, we'll use this location rather than
* the one currently in <code>.META.</code>
* @throws IOException if a remote or network exception occurs
*/
public void closeRegion(final byte [] regionname, final Object... args)
public void closeRegion(final byte [] regionname, final String hostAndPort)
throws IOException {
// Be careful. Must match the handler over in HMaster at MODIFY_CLOSE_REGION
int len = (args == null)? 0: args.length;
int xtraArgsCount = 1;
Object [] newargs = new Object[len + xtraArgsCount];
newargs[0] = regionname;
if(args != null) {
System.arraycopy(args, 0, newargs, xtraArgsCount, len);
if (hostAndPort != null) {
HServerAddress hsa = new HServerAddress(hostAndPort);
Pair<HRegionInfo, HServerAddress> pair =
MetaReader.getRegion(getCatalogTracker(), regionname);
closeRegion(hsa, pair.getFirst());
} else {
Pair<HRegionInfo, HServerAddress> pair =
MetaReader.getRegion(getCatalogTracker(), regionname);
closeRegion(pair.getSecond(), pair.getFirst());
}
modifyTable(HConstants.META_TABLE_NAME, HConstants.Modify.CLOSE_REGION,
newargs);
}
private void closeRegion(final HServerAddress hsa, final HRegionInfo hri)
throws IOException {
HRegionInterface rs = this.connection.getHRegionConnection(hsa);
rs.closeRegion(hri);
}
/**
@ -700,7 +756,25 @@ public class HBaseAdmin {
* @throws IOException if a remote or network exception occurs
*/
public void flush(final byte [] tableNameOrRegionName) throws IOException {
modifyTable(tableNameOrRegionName, HConstants.Modify.TABLE_FLUSH);
boolean isRegionName = isRegionName(tableNameOrRegionName);
if (isRegionName) {
Pair<HRegionInfo, HServerAddress> pair =
MetaReader.getRegion(getCatalogTracker(), tableNameOrRegionName);
flush(pair.getSecond(), pair.getFirst());
} else {
List<Pair<HRegionInfo, HServerAddress>> pairs =
MetaReader.getTableRegionsAndLocations(getCatalogTracker(),
Bytes.toString(tableNameOrRegionName));
for (Pair<HRegionInfo, HServerAddress> pair: pairs) {
flush(pair.getSecond(), pair.getFirst());
}
}
}
private void flush(final HServerAddress hsa, final HRegionInfo hri)
throws IOException {
HRegionInterface rs = this.connection.getHRegionConnection(hsa);
rs.flushRegion(hri);
}
/**
@ -722,7 +796,7 @@ public class HBaseAdmin {
* @throws IOException if a remote or network exception occurs
*/
public void compact(final byte [] tableNameOrRegionName) throws IOException {
modifyTable(tableNameOrRegionName, HConstants.Modify.TABLE_COMPACT);
compact(tableNameOrRegionName, false);
}
/**
@ -746,7 +820,63 @@ public class HBaseAdmin {
*/
public void majorCompact(final byte [] tableNameOrRegionName)
throws IOException {
modifyTable(tableNameOrRegionName, HConstants.Modify.TABLE_MAJOR_COMPACT);
compact(tableNameOrRegionName, true);
}
/**
* Compact a table or an individual region.
* Asynchronous operation.
*
* @param tableNameOrRegionName table or region to compact
* @param major True if we are to do a major compaction.
* @throws IOException if a remote or network exception occurs
*/
private void compact(final byte [] tableNameOrRegionName, final boolean major)
throws IOException {
if (isRegionName(tableNameOrRegionName)) {
Pair<HRegionInfo, HServerAddress> pair =
MetaReader.getRegion(getCatalogTracker(), tableNameOrRegionName);
compact(pair.getSecond(), pair.getFirst(), major);
} else {
List<Pair<HRegionInfo, HServerAddress>> pairs =
MetaReader.getTableRegionsAndLocations(getCatalogTracker(),
Bytes.toString(tableNameOrRegionName));
for (Pair<HRegionInfo, HServerAddress> pair: pairs) {
compact(pair.getSecond(), pair.getFirst(), major);
}
}
}
private void compact(final HServerAddress hsa, final HRegionInfo hri,
final boolean major)
throws IOException {
HRegionInterface rs = this.connection.getHRegionConnection(hsa);
rs.compactRegion(hri, major);
}
/**
* Move the region <code>r</code> to <code>dest</code>.
* @param encodedRegionName The encoded region name.
* @param destServerName The servername of the destination regionserver
* @throws UnknownRegionException Thrown if we can't find a region named
* <code>encodedRegionName</code>
* @throws ZooKeeperConnectionException
* @throws MasterNotRunningException
*/
public void move(final byte [] encodedRegionName, final byte [] destServerName)
throws UnknownRegionException, MasterNotRunningException, ZooKeeperConnectionException {
getMaster().move(encodedRegionName, destServerName);
}
/**
* @param b If true, enable balancer. If false, disable balancer.
* @return Previous balancer value
* @throws ZooKeeperConnectionException
* @throws MasterNotRunningException
*/
public boolean balance(final boolean b)
throws MasterNotRunningException, ZooKeeperConnectionException {
return getMaster().balance(b);
}
/**
@ -768,32 +898,31 @@ public class HBaseAdmin {
* @throws IOException if a remote or network exception occurs
*/
public void split(final byte [] tableNameOrRegionName) throws IOException {
modifyTable(tableNameOrRegionName, HConstants.Modify.TABLE_SPLIT);
if (isRegionName(tableNameOrRegionName)) {
// Its a possible region name.
Pair<HRegionInfo, HServerAddress> pair =
MetaReader.getRegion(getCatalogTracker(), tableNameOrRegionName);
split(pair.getSecond(), pair.getFirst());
} else {
List<Pair<HRegionInfo, HServerAddress>> pairs =
MetaReader.getTableRegionsAndLocations(getCatalogTracker(),
Bytes.toString(tableNameOrRegionName));
for (Pair<HRegionInfo, HServerAddress> pair: pairs) {
split(pair.getSecond(), pair.getFirst());
}
}
}
/*
* Call modifyTable using passed tableName or region name String. If no
* such table, presume we have been passed a region name.
* @param tableNameOrRegionName
* @param op
* @throws IOException
*/
private void modifyTable(final byte [] tableNameOrRegionName,
final HConstants.Modify op)
private void split(final HServerAddress hsa, final HRegionInfo hri)
throws IOException {
if (tableNameOrRegionName == null) {
throw new IllegalArgumentException("Pass a table name or region name");
}
byte [] tableName = tableExists(tableNameOrRegionName)?
tableNameOrRegionName: null;
byte [] regionName = tableName == null? tableNameOrRegionName: null;
Object [] args = regionName == null? null: new byte [][] {regionName};
modifyTable(tableName == null? null: tableName, op, args);
HRegionInterface rs = this.connection.getHRegionConnection(hsa);
rs.splitRegion(hri);
}
/**
* Modify an existing table, more IRB friendly version.
* Asynchronous operation.
* Asynchronous operation. This means that it may be a while before your
* schema change is updated across all of the table.
*
* @param tableName name of table.
* @param htd modified description of the table
@ -801,107 +930,57 @@ public class HBaseAdmin {
*/
public void modifyTable(final byte [] tableName, HTableDescriptor htd)
throws IOException {
modifyTable(tableName, HConstants.Modify.TABLE_SET_HTD, htd);
}
/**
* Modify an existing table.
* Asynchronous operation.
*
* @param tableName name of table. May be null if we are operating on a
* region.
* @param op table modification operation
* @param args operation specific arguments
* @throws IOException if a remote or network exception occurs
*/
public void modifyTable(final byte [] tableName, HConstants.Modify op,
Object... args)
throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
// Let pass if its a catalog table. Used by admins.
if (tableName != null && !MetaUtils.isMetaTableName(tableName)) {
// This will throw exception
HTableDescriptor.isLegalTableName(tableName);
}
Writable[] arr = null;
try {
switch (op) {
case TABLE_SET_HTD:
if (args == null || args.length < 1 ||
!(args[0] instanceof HTableDescriptor)) {
throw new IllegalArgumentException("SET_HTD requires a HTableDescriptor");
}
arr = new Writable[1];
arr[0] = (HTableDescriptor)args[0];
this.master.modifyTable(tableName, op, arr);
break;
case TABLE_COMPACT:
case TABLE_SPLIT:
case TABLE_MAJOR_COMPACT:
case TABLE_FLUSH:
if (args != null && args.length > 0) {
arr = new Writable[1];
if (args[0] instanceof byte[]) {
arr[0] = new ImmutableBytesWritable((byte[])args[0]);
} else if (args[0] instanceof ImmutableBytesWritable) {
arr[0] = (ImmutableBytesWritable)args[0];
} else if (args[0] instanceof String) {
arr[0] = new ImmutableBytesWritable(Bytes.toBytes((String)args[0]));
} else {
throw new IllegalArgumentException("Requires byte[], String, or" +
"ImmutableBytesWritable");
}
}
this.master.modifyTable(tableName, op, arr);
break;
case CLOSE_REGION:
if (args == null || args.length < 1) {
throw new IllegalArgumentException("Requires at least a region name");
}
arr = new Writable[args.length];
for (int i = 0; i < args.length; i++) {
if (args[i] instanceof byte[]) {
arr[i] = new ImmutableBytesWritable((byte[])args[i]);
} else if (args[i] instanceof ImmutableBytesWritable) {
arr[i] = (ImmutableBytesWritable)args[i];
} else if (args[i] instanceof String) {
arr[i] = new ImmutableBytesWritable(Bytes.toBytes((String)args[i]));
} else if (args[i] instanceof Boolean) {
arr[i] = new BooleanWritable((Boolean) args[i]);
} else {
throw new IllegalArgumentException("Requires byte [] or " +
"ImmutableBytesWritable, not " + args[i]);
}
}
this.master.modifyTable(tableName, op, arr);
break;
default:
throw new IOException("unknown modifyTable op " + op);
}
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
getMaster().modifyTable(tableName, htd);
} catch (RemoteException re) {
// Convert RE exceptions in here; client shouldn't have to deal with them,
// at least w/ the type of exceptions that come out of this method:
// TableNotFoundException, etc.
throw RemoteExceptionHandler.decodeRemoteException(re);
}
}
/**
* Shuts down the HBase instance
* @param tableNameOrRegionName Name of a table or name of a region.
* @return True if <code>tableNameOrRegionName</code> is *possibly* a region
* name else false if a verified tablename (we call {@link #tableExists(byte[])};
* else we throw an exception.
* @throws ZooKeeperConnectionException
* @throws MasterNotRunningException
*/
private boolean isRegionName(final byte [] tableNameOrRegionName)
throws MasterNotRunningException, ZooKeeperConnectionException {
if (tableNameOrRegionName == null) {
throw new IllegalArgumentException("Pass a table name or region name");
}
return !tableExists(tableNameOrRegionName);
}
/**
* Shuts down the HBase cluster
* @throws IOException if a remote or network exception occurs
*/
public synchronized void shutdown() throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
isMasterRunning();
try {
this.master.shutdown();
getMaster().shutdown();
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
}
}
/**
* Shuts down the current HBase master only.
* Does not shutdown the cluster.
* @see #shutdown()
* @throws IOException if a remote or network exception occurs
*/
public synchronized void stopMaster() throws IOException {
isMasterRunning();
try {
getMaster().stopMaster();
} catch (RemoteException e) {
throw RemoteExceptionHandler.decodeRemoteException(e);
} finally {
this.master = null;
}
}
@ -910,10 +989,7 @@ public class HBaseAdmin {
* @throws IOException if a remote or network exception occurs
*/
public ClusterStatus getClusterStatus() throws IOException {
if (this.master == null) {
throw new MasterNotRunningException("master has been shut down");
}
return this.master.getClusterStatus();
return getMaster().getClusterStatus();
}
private HRegionLocation getFirstMetaServerForTable(final byte [] tableName)
@ -926,12 +1002,13 @@ public class HBaseAdmin {
* Check to see if HBase is running. Throw an exception if not.
*
* @param conf system configuration
* @throws MasterNotRunningException if a remote or network exception occurs
* @throws MasterNotRunningException if the master is not running
* @throws ZooKeeperConnectionException if unable to connect to zookeeper
*/
public static void checkHBaseAvailable(Configuration conf)
throws MasterNotRunningException {
throws MasterNotRunningException, ZooKeeperConnectionException {
Configuration copyOfConf = HBaseConfiguration.create(conf);
copyOfConf.setInt("hbase.client.retries.number", 1);
new HBaseAdmin(copyOfConf);
}
}
}

9
src/main/java/org/apache/hadoop/hbase/client/HBaseFsck.java
View File

@ -23,6 +23,7 @@ import java.io.IOException;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.NavigableSet;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.concurrent.atomic.AtomicInteger;
@ -42,6 +43,7 @@ import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.client.MetaScanner.MetaScannerVisitor;
import org.apache.hadoop.hbase.ipc.HMasterInterface;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
@ -73,9 +75,10 @@ public class HBaseFsck extends HBaseAdmin {
*
* @param conf Configuration object
* @throws MasterNotRunningException if the master is not running
* @throws ZooKeeperConnectionException if unable to connect to zookeeper
*/
public HBaseFsck(Configuration conf)
throws MasterNotRunningException, IOException {
throws MasterNotRunningException, ZooKeeperConnectionException, IOException {
super(conf);
this.conf = conf;
@ -261,10 +264,10 @@ public class HBaseFsck extends HBaseAdmin {
rsinfo.getServerAddress());
// list all online regions from this region server
HRegionInfo[] regions = server.getRegionsAssignment();
NavigableSet<HRegionInfo> regions = server.getOnlineRegions();
if (details) {
System.out.print("\nRegionServer:" + rsinfo.getServerName() +
" number of regions:" + regions.length);
" number of regions:" + regions.size());
for (HRegionInfo rinfo: regions) {
System.out.print("\n\t name:" + rinfo.getRegionNameAsString() +
" id:" + rinfo.getRegionId() +

91
src/main/java/org/apache/hadoop/hbase/client/HConnection.java
View File

@ -19,50 +19,55 @@
*/
package org.apache.hadoop.hbase.client;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HRegionLocation;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.ipc.HMasterInterface;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HRegionLocation;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.ipc.HMasterInterface;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher;
/**
* Cluster connection.
* {@link HConnectionManager} manages instances of this class.
*/
public interface HConnection {
/**
* Retrieve ZooKeeperWrapper used by the connection.
* @return ZooKeeperWrapper handle being used by the connection.
* Retrieve ZooKeeperWatcher used by the connection.
* @return ZooKeeperWatcher handle being used by the connection.
* @throws IOException if a remote or network exception occurs
*/
public ZooKeeperWrapper getZooKeeperWrapper() throws IOException;
public ZooKeeperWatcher getZooKeeperWatcher() throws IOException;
/**
* @return proxy connection to master server for this instance
* @throws MasterNotRunningException if the master is not running
* @throws ZooKeeperConnectionException if unable to connect to zookeeper
*/
public HMasterInterface getMaster() throws MasterNotRunningException;
public HMasterInterface getMaster()
throws MasterNotRunningException, ZooKeeperConnectionException;
/** @return - true if the master server is running */
public boolean isMasterRunning();
public boolean isMasterRunning()
throws MasterNotRunningException, ZooKeeperConnectionException;
/**
* Checks if <code>tableName</code> exists.
* @param tableName Table to check.
* @return True if table exists already.
* @throws MasterNotRunningException if the master is not running
* @throws ZooKeeperConnectionException if unable to connect to zookeeper
*/
public boolean tableExists(final byte [] tableName)
throws MasterNotRunningException;
throws MasterNotRunningException, ZooKeeperConnectionException;
/**
* A table that isTableEnabled == false and isTableDisabled == false
@ -131,7 +136,7 @@ public interface HConnection {
* lives in, ignoring any value that might be in the cache.
* @param tableName name of the table <i>row</i> is in
* @param row row key you're trying to find the region of
* @return HRegionLocation that describes where to find the reigon in
* @return HRegionLocation that describes where to find the region in
* question
* @throws IOException if a remote or network exception occurs
*/
@ -139,6 +144,25 @@ public interface HConnection {
final byte [] row)
throws IOException;
/**
* Gets the location of the region of <i>regionName</i>.
* @param regionName name of the region to locate
* @return HRegionLocation that describes where to find the region in
* question
* @throws IOException if a remote or network exception occurs
*/
public HRegionLocation locateRegion(final byte [] regionName)
throws IOException;
/**
* Gets the locations of all regions in the specified table, <i>tableName</i>.
* @param tableName table to get regions of
* @return list of region locations for all regions of table
* @throws IOException
*/
public List<HRegionLocation> locateRegions(byte[] tableName)
throws IOException;
/**
* Establishes a connection to the region server at the specified address.
* @param regionServer - the server to connect to
@ -197,21 +221,6 @@ public interface HConnection {
public <T> T getRegionServerWithoutRetries(ServerCallable<T> callable)
throws IOException, RuntimeException;
/**
* Process a mixed batch of Get, Put and Delete actions. All actions for a
* RegionServer are forwarded in one RPC call.
*
* @param actions The collection of actions.
* @param tableName Name of the hbase table
* @param pool thread pool for parallel execution
* @param results An empty array, same size as list. If an exception is thrown,
* you can test here for partial results, and to determine which actions
* processed successfully.
* @throws IOException
*/
public void processBatch(List<Row> actions, final byte[] tableName,
ExecutorService pool, Result[] results)
throws IOException;
/**
* Process a batch of Puts. Does the retries.
@ -219,32 +228,20 @@ public interface HConnection {
* @param tableName The name of the table
* @return Count of committed Puts. On fault, < list.size().
* @throws IOException if a remote or network exception occurs
* @deprecated Use HConnectionManager::processBatch instead.
*/
public int processBatchOfRows(ArrayList<Put> list, byte[] tableName, ExecutorService pool)
public int processBatchOfRows(ArrayList<Put> list, byte[] tableName)
throws IOException;
/**
* Process a batch of Deletes. Does the retries.
* @param list A batch of Deletes to process.
* @param tableName The name of the table
* @return Count of committed Deletes. On fault, < list.size().
* @param tableName The name of the table
* @throws IOException if a remote or network exception occurs
* @deprecated Use HConnectionManager::processBatch instead.
*/
public int processBatchOfDeletes(List<Delete> list, byte[] tableName, ExecutorService pool)
public int processBatchOfDeletes(List<Delete> list, byte[] tableName)
throws IOException;
/**
* Process a batch of Puts.
*
* @param list The collection of actions. The list is mutated: all successful Puts
* are removed from the list.
* @param tableName Name of the hbase table
* @param pool Thread pool for parallel execution
* @throws IOException
* @deprecated Use HConnectionManager::processBatch instead.
*/
public void processBatchOfPuts(List<Put> list,
final byte[] tableName, ExecutorService pool) throws IOException;
@ -261,7 +258,7 @@ public interface HConnection {
/**
* Check whether region cache prefetch is enabled or not.
* @param tableName name of table to check
* @return true if table's region cache prefetch is enabled. Otherwise
* @return true if table's region cache prefecth is enabled. Otherwise
* it is disabled.
*/
public boolean getRegionCachePrefetch(final byte[] tableName);

818
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
View File

File diff suppressed because it is too large Load Diff

118
src/main/java/org/apache/hadoop/hbase/client/HTable.java
View File

@ -32,6 +32,7 @@ import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.NotServingRegionException;
import org.apache.hadoop.hbase.UnknownScannerException;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.client.MetaScanner.MetaScannerVisitor;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Pair;
@ -45,7 +46,6 @@ import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadFactory;
@ -77,7 +77,7 @@ public class HTable implements HTableInterface {
private long currentWriteBufferSize;
protected int scannerCaching;
private int maxKeyValueSize;
private ExecutorService pool; // For Multi
private long maxScannerResultSize;
/**
@ -144,11 +144,13 @@ public class HTable implements HTableInterface {
HConstants.HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY,
HConstants.DEFAULT_HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE);
this.maxKeyValueSize = conf.getInt("hbase.client.keyvalue.maxsize", -1);
int nrThreads = conf.getInt("hbase.htable.threads.max", getCurrentNrHRS());
if (nrThreads == 0) {
nrThreads = 1; // is there a better default?
int nrHRS = getCurrentNrHRS();
if (nrHRS == 0) {
// No servers running -- set default of 10 threads.
nrHRS = 10;
}
int nrThreads = conf.getInt("hbase.htable.threads.max", nrHRS);
// Unfortunately Executors.newCachedThreadPool does not allow us to
// set the maximum size of the pool, so we have to do it ourselves.
@ -169,12 +171,13 @@ public class HTable implements HTableInterface {
* @throws IOException if a remote or network exception occurs
*/
int getCurrentNrHRS() throws IOException {
return HConnectionManager
.getClientZooKeeperWatcher(this.configuration)
.getZooKeeperWrapper()
.getRSDirectoryCount();
HBaseAdmin admin = new HBaseAdmin(this.configuration);
return admin.getClusterStatus().getServers();
}
// For multiput
private ExecutorService pool;
/**
* Tells whether or not a table is enabled or not.
* @param tableName Name of table to check.
@ -505,40 +508,6 @@ public class HTable implements HTableInterface {
);
}
/**
* Method that does a batch call on Deletes, Gets and Puts.
*
* @param actions list of Get, Put, Delete objects
* @param results Empty Result[], same size as actions. Provides access to partial
* results, in case an exception is thrown. A null in the result array means that
* the call for that action failed, even after retries
* @throws IOException
*/
public synchronized void batch(final List<Row> actions, final Result[] results) throws IOException {
connection.processBatch(actions, tableName, pool, results);
}
/**
* Method that does a batch call on Deletes, Gets and Puts.
*
* @param actions list of Get, Put, Delete objects
* @return the results from the actions. A null in the return array means that
* the call for that action failed, even after retries
* @throws IOException
*/
public synchronized Result[] batch(final List<Row> actions) throws IOException {
Result[] results = new Result[actions.size()];
connection.processBatch(actions, tableName, pool, results);
return results;
}
/**
* Deletes the specified cells/row.
*
* @param delete The object that specifies what to delete.
* @throws IOException if a remote or network exception occurs.
* @since 0.20.0
*/
public void delete(final Delete delete)
throws IOException {
connection.getRegionServerWithRetries(
@ -551,28 +520,13 @@ public class HTable implements HTableInterface {
);
}
/**
* Deletes the specified cells/rows in bulk.
* @param deletes List of things to delete. As a side effect, it will be modified:
* successful {@link Delete}s are removed. The ordering of the list will not change.
* @throws IOException if a remote or network exception occurs. In that case
* the {@code deletes} argument will contain the {@link Delete} instances
* that have not be successfully applied.
* @since 0.20.1
*/
public void delete(final List<Delete> deletes)
throws IOException {
Result[] results = new Result[deletes.size()];
connection.processBatch((List) deletes, tableName, pool, results);
// mutate list so that it is empty for complete success, or contains only failed records
// results are returned in the same order as the requests in list
// walk the list backwards, so we can remove from list without impacting the indexes of earlier members
for (int i = results.length - 1; i>=0; i--) {
// if result is not null, it succeeded
if (results[i] != null) {
deletes.remove(i);
}
int last = 0;
try {
last = connection.processBatchOfDeletes(deletes, this.tableName);
} finally {
deletes.subList(0, last).clear();
}
}
@ -601,7 +555,6 @@ public class HTable implements HTableInterface {
return incrementColumnValue(row, family, qualifier, amount, true);
}
@SuppressWarnings({"ThrowableInstanceNeverThrown"})
public long incrementColumnValue(final byte [] row, final byte [] family,
final byte [] qualifier, final long amount, final boolean writeToWAL)
throws IOException {
@ -675,7 +628,7 @@ public class HTable implements HTableInterface {
public Boolean call() throws IOException {
return server.checkAndDelete(
location.getRegionInfo().getRegionName(),
row, family, qualifier, value, delete)
row, family, qualifier, value, delete)
? Boolean.TRUE : Boolean.FALSE;
}
}
@ -704,17 +657,10 @@ public class HTable implements HTableInterface {
);
}
/**
* Executes all the buffered {@link Put} operations.
* <p>
* This method gets called once automatically for every {@link Put} or batch
* of {@link Put}s (when {@link #batch(List)} is used) when
* {@link #isAutoFlush()} is {@code true}.
* @throws IOException if a remote or network exception occurs.
*/
public void flushCommits() throws IOException {
try {
connection.processBatchOfPuts(writeBuffer, tableName, pool);
connection.processBatchOfPuts(writeBuffer,
tableName, pool);
} finally {
// the write buffer was adjusted by processBatchOfPuts
currentWriteBufferSize = 0;
@ -1153,10 +1099,12 @@ public class HTable implements HTableInterface {
Thread t = new Thread(group, r,
namePrefix + threadNumber.getAndIncrement(),
0);
if (!t.isDaemon())
t.setDaemon(true);
if (t.getPriority() != Thread.NORM_PRIORITY)
t.setPriority(Thread.NORM_PRIORITY);
if (!t.isDaemon()) {
t.setDaemon(true);
}
if (t.getPriority() != Thread.NORM_PRIORITY) {
t.setPriority(Thread.NORM_PRIORITY);
}
return t;
}
}
@ -1168,9 +1116,10 @@ public class HTable implements HTableInterface {
* @param tableName name of table to configure.
* @param enable Set to true to enable region cache prefetch. Or set to
* false to disable it.
* @throws ZooKeeperConnectionException
*/
public static void setRegionCachePrefetch(final byte[] tableName,
boolean enable) {
boolean enable) throws ZooKeeperConnectionException {
HConnectionManager.getConnection(HBaseConfiguration.create()).
setRegionCachePrefetch(tableName, enable);
}
@ -1183,9 +1132,10 @@ public class HTable implements HTableInterface {
* @param tableName name of table to configure.
* @param enable Set to true to enable region cache prefetch. Or set to
* false to disable it.
* @throws ZooKeeperConnectionException
*/
public static void setRegionCachePrefetch(final Configuration conf,
final byte[] tableName, boolean enable) {
final byte[] tableName, boolean enable) throws ZooKeeperConnectionException {
HConnectionManager.getConnection(conf).setRegionCachePrefetch(
tableName, enable);
}
@ -1196,9 +1146,10 @@ public class HTable implements HTableInterface {
* @param tableName name of table to check
* @return true if table's region cache prefecth is enabled. Otherwise
* it is disabled.
* @throws ZooKeeperConnectionException
*/
public static boolean getRegionCachePrefetch(final Configuration conf,
final byte[] tableName) {
final byte[] tableName) throws ZooKeeperConnectionException {
return HConnectionManager.getConnection(conf).getRegionCachePrefetch(
tableName);
}
@ -1208,8 +1159,9 @@ public class HTable implements HTableInterface {
* @param tableName name of table to check
* @return true if table's region cache prefecth is enabled. Otherwise
* it is disabled.
* @throws ZooKeeperConnectionException
*/
public static boolean getRegionCachePrefetch(final byte[] tableName) {
public static boolean getRegionCachePrefetch(final byte[] tableName) throws ZooKeeperConnectionException {
return HConnectionManager.getConnection(HBaseConfiguration.create()).
getRegionCachePrefetch(tableName);
}

35
src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
View File

@ -20,6 +20,10 @@
package org.apache.hadoop.hbase.client;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
@ -27,8 +31,6 @@ import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Writables;
import java.io.IOException;
/**
* Scanner class that contains the <code>.META.</code> table scanning logic
* and uses a Retryable scanner. Provided visitors will be called
@ -38,7 +40,6 @@ import java.io.IOException;
* minor releases.
*/
public class MetaScanner {
/**
* Scans the meta table and calls a visitor on each RowResult and uses a empty
* start row value as table name.
@ -50,7 +51,7 @@ public class MetaScanner {
public static void metaScan(Configuration configuration,
MetaScannerVisitor visitor)
throws IOException {
metaScan(configuration, visitor, HConstants.EMPTY_START_ROW);
metaScan(configuration, visitor, null);
}
/**
@ -169,6 +170,32 @@ public class MetaScanner {
} while (Bytes.compareTo(startRow, HConstants.LAST_ROW) != 0);
}
/**
* Lists all of the regions currently in META.
* @return
* @throws IOException
*/
public static List<HRegionInfo> listAllRegions(Configuration conf)
throws IOException {
final List<HRegionInfo> regions = new ArrayList<HRegionInfo>();
MetaScannerVisitor visitor =
new MetaScannerVisitor() {
@Override
public boolean processRow(Result result) throws IOException {
if (result == null || result.isEmpty()) {
return true;
}
HRegionInfo regionInfo = Writables.getHRegionInfo(
result.getValue(HConstants.CATALOG_FAMILY,
HConstants.REGIONINFO_QUALIFIER));
regions.add(regionInfo);
return true;
}
};
metaScan(conf, visitor);
return regions;
}
/**
* Visitor class called to process each row of the .META. table
*/

1
src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
View File

@ -34,7 +34,6 @@ import java.util.Map;
import java.util.TreeMap;
/**
* @deprecated Use MultiAction instead
* Data type class for putting multiple regions worth of puts in one RPC.
*/
public class MultiPut implements Writable {

1
src/main/java/org/apache/hadoop/hbase/client/MultiPutResponse.java
View File

@ -30,7 +30,6 @@ import java.util.Map;
import java.util.TreeMap;
/**
* @deprecated Replaced by MultiResponse
* Response class for MultiPut.
*/
public class MultiPutResponse implements Writable {

4
src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedException.java
View File

@ -31,6 +31,10 @@ public class RetriesExhaustedException extends IOException {
super(msg);
}
public RetriesExhaustedException(final String msg, final IOException e) {
super(msg, e);
}
/**
* Create a new RetriesExhaustedException from the list of prior failures.
* @param serverName name of HRegionServer

4
src/main/java/org/apache/hadoop/hbase/client/ServerConnectionManager.java
View File

@ -21,6 +21,7 @@
package org.apache.hadoop.hbase.client;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
/**
@ -38,8 +39,9 @@ public class ServerConnectionManager extends HConnectionManager {
* If no current connection exists, create a new connection for that instance
* @param conf configuration
* @return HConnection object for the instance specified by the configuration
* @throws ZooKeeperConnectionException
*/
public static ServerConnection getConnection(Configuration conf) {
public static ServerConnection getConnection(Configuration conf) throws ZooKeeperConnectionException {
return (ServerConnection) HConnectionManager.getConnection(conf);
}
}

223
src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java Normal file
View File

@ -0,0 +1,223 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.executor;
import java.io.IOException;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.Server;
/**
* Abstract base class for all HBase event handlers. Subclasses should
* implement the {@link #process()} method. Subclasses should also do all
* necessary checks up in their constructor if possible -- check table exists,
* is disabled, etc. -- so they fail fast rather than later when process is
* running. Do it this way because process be invoked directly but event
* handlers are also
* run in an executor context -- i.e. asynchronously -- and in this case,
* exceptions thrown at process time will not be seen by the invoker, not till
* we implement a call-back mechanism so the client can pick them up later.
* <p>
* Event handlers have an {@link EventType}.
* {@link EventType} is a list of ALL handler event types. We need to keep
* a full list in one place -- and as enums is a good shorthand for an
* implemenations -- because event handlers can be passed to executors when
* they are to be run asynchronously. The
* hbase executor, see {@link ExecutorService}, has a switch for passing
* event type to executor.
* <p>
* Event listeners can be installed and will be called pre- and post- process if
* this EventHandler is run in a Thread (its a Runnable so if its {@link #run()}
* method gets called). Implement
* {@link EventHandlerListener}s, and registering using
* {@link #setListener(EventHandlerListener)}.
* @see {@link ExecutorService}
*/
public abstract class EventHandler implements Runnable, Comparable<Runnable> {
private static final Log LOG = LogFactory.getLog(EventHandler.class);
// type of event this object represents
protected EventType eventType;
protected Server server;
// sequence id generator for default FIFO ordering of events
protected static AtomicLong seqids = new AtomicLong(0);
// sequence id for this event
private final long seqid;
// Listener to call pre- and post- processing. May be null.
private EventHandlerListener listener;
/**
* This interface provides pre- and post-process hooks for events.
*/
public interface EventHandlerListener {
/**
* Called before any event is processed
* @param The event handler whose process method is about to be called.
*/
public void beforeProcess(EventHandler event);
/**
* Called after any event is processed
* @param The event handler whose process method is about to be called.
*/
public void afterProcess(EventHandler event);
}
/**
* List of all HBase event handler types. Event types are named by a
* convention: event type names specify the component from which the event
* originated and then where its destined -- e.g. RS2ZK_ prefix means the
* event came from a regionserver destined for zookeeper -- and then what
* the even is; e.g. REGION_OPENING.
*
* <p>We give the enums indices so we can add types later and keep them
* grouped together rather than have to add them always to the end as we
* would have to if we used raw enum ordinals.
*/
public enum EventType {
// Messages originating from RS (NOTE: there is NO direct communication from
// RS to Master). These are a result of RS updates into ZK.
RS2ZK_REGION_CLOSING (1), // RS is in process of closing a region
RS2ZK_REGION_CLOSED (2), // RS has finished closing a region
RS2ZK_REGION_OPENING (3), // RS is in process of opening a region
RS2ZK_REGION_OPENED (4), // RS has finished opening a region
// Messages originating from Master to RS
M2RS_OPEN_REGION (20), // Master asking RS to open a region
M2RS_OPEN_ROOT (21), // Master asking RS to open root
M2RS_OPEN_META (22), // Master asking RS to open meta
M2RS_CLOSE_REGION (23), // Master asking RS to close a region
M2RS_CLOSE_ROOT (24), // Master asking RS to close root
M2RS_CLOSE_META (25), // Master asking RS to close meta
// Messages originating from Client to Master
C2M_DELETE_TABLE (40), // Client asking Master to delete a table
C2M_DISABLE_TABLE (41), // Client asking Master to disable a table
C2M_ENABLE_TABLE (42), // Client asking Master to enable a table
C2M_MODIFY_TABLE (43), // Client asking Master to modify a table
C2M_ADD_FAMILY (44), // Client asking Master to add family to table
C2M_DELETE_FAMILY (45), // Client asking Master to delete family of table
C2M_MODIFY_FAMILY (46), // Client asking Master to modify family of table
// Updates from master to ZK. This is done by the master and there is
// nothing to process by either Master or RS
M2ZK_REGION_OFFLINE (50), // Master adds this region as offline in ZK
// Master controlled events to be executed on the master
M_SERVER_SHUTDOWN (70); // Master is processing shutdown of a RS
/**
* Constructor
*/
EventType(int value) {}
}
/**
* Default base class constructor.
*/
public EventHandler(Server server, EventType eventType) {
this.server = server;
this.eventType = eventType;
seqid = seqids.incrementAndGet();
}
public void run() {
try {
if (getListener() != null) getListener().beforeProcess(this);
process();
if (getListener() != null) getListener().afterProcess(this);
} catch(Throwable t) {
LOG.error("Caught throwable while processing event " + eventType, t);
}
}
/**
* This method is the main processing loop to be implemented by the various
* subclasses.
* @throws IOException
*/
public abstract void process() throws IOException;
/**
* Return the event type
* @return
*/
public EventType getEventType() {
return this.eventType;
}
/**
* Get the priority level for this handler instance. This uses natural
* ordering so lower numbers are higher priority.
* <p>
* Lowest priority is Integer.MAX_VALUE. Highest priority is 0.
* <p>
* Subclasses should override this method to allow prioritizing handlers.
* <p>
* Handlers with the same priority are handled in FIFO order.
* <p>
* @return Integer.MAX_VALUE by default, override to set higher priorities
*/
public int getPriority() {
return Integer.MAX_VALUE;
}
/**
* @return This events' sequence id.
*/
public long getSeqid() {
return this.seqid;
}
/**
* Default prioritized runnable comparator which implements a FIFO ordering.
* <p>
* Subclasses should not override this. Instead, if they want to implement
* priority beyond FIFO, they should override {@link #getPriority()}.
*/
@Override
public int compareTo(Runnable o) {
EventHandler eh = (EventHandler)o;
if(getPriority() != eh.getPriority()) {
return (getPriority() < eh.getPriority()) ? -1 : 1;
}
return (this.seqid < eh.seqid) ? -1 : 1;
}
/**
* @return Current listener or null if none set.
*/
public synchronized EventHandlerListener getListener() {
return listener;
}
/**
* @param listener Listener to call pre- and post- {@link #process()}.
*/
public synchronized void setListener(EventHandlerListener listener) {
this.listener = listener;
}
}

285
src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java Normal file
View File

@ -0,0 +1,285 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.executor;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.PriorityBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.executor.EventHandler.EventHandlerListener;
import org.apache.hadoop.hbase.executor.EventHandler.EventType;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
/**
* This is a generic executor service. This component abstracts a
* threadpool, a queue to which {@link EventHandler.EventType}s can be submitted,
* and a <code>Runnable</code> that handles the object that is added to the queue.
*
* <p>In order to create a new service, create an instance of this class and
* then do: <code>instance.startExecutorService("myService");</code>. When done
* call {@link #shutdown()}.
*
* <p>In order to use the service created above, call
* {@link #submit(EventHandler)}. Register pre- and post- processing listeners
* by registering your implementation of {@link EventHandler.EventHandlerListener}
* with {@link #registerListener(EventType, EventHandlerListener)}. Be sure
* to deregister your listener when done via {@link #unregisterListener(EventType)}.
*/
public class ExecutorService {
private static final Log LOG = LogFactory.getLog(ExecutorService.class);
// hold the all the executors created in a map addressable by their names
private final ConcurrentHashMap<String, Executor> executorMap =
new ConcurrentHashMap<String, Executor>();
// listeners that are called before and after an event is processed
private ConcurrentHashMap<EventHandler.EventType, EventHandlerListener> eventHandlerListeners =
new ConcurrentHashMap<EventHandler.EventType, EventHandlerListener>();
// Name of the server hosting this executor service.
private final String servername;
/**
* The following is a list of all executor types, both those that run in the
* master and those that run in the regionserver.
*/
public enum ExecutorType {
// Master executor services
MASTER_CLOSE_REGION (1),
MASTER_OPEN_REGION (2),
MASTER_SERVER_OPERATIONS (3),
MASTER_TABLE_OPERATIONS (4),
MASTER_RS_SHUTDOWN (5),
// RegionServer executor services
RS_OPEN_REGION (20),
RS_OPEN_ROOT (21),
RS_OPEN_META (22),
RS_CLOSE_REGION (23),
RS_CLOSE_ROOT (24),
RS_CLOSE_META (25);
ExecutorType(int value) {}
/**
* @param serverName
* @return Conflation of the executor type and the passed servername.
*/
String getExecutorName(String serverName) {
return this.toString() + "-" + serverName;
}
}
/**
* Returns the executor service type (the thread pool instance) for the
* passed event handler type.
* @param type EventHandler type.
*/
public ExecutorType getExecutorServiceType(final EventHandler.EventType type) {
switch(type) {
// Master executor services
case RS2ZK_REGION_CLOSED:
return ExecutorType.MASTER_CLOSE_REGION;
case RS2ZK_REGION_OPENED:
return ExecutorType.MASTER_OPEN_REGION;
case M_SERVER_SHUTDOWN:
return ExecutorType.MASTER_SERVER_OPERATIONS;
case C2M_DELETE_TABLE:
case C2M_DISABLE_TABLE:
case C2M_ENABLE_TABLE:
case C2M_MODIFY_TABLE:
return ExecutorType.MASTER_TABLE_OPERATIONS;
// RegionServer executor services
case M2RS_OPEN_REGION:
return ExecutorType.RS_OPEN_REGION;
case M2RS_OPEN_ROOT:
return ExecutorType.RS_OPEN_ROOT;
case M2RS_OPEN_META:
return ExecutorType.RS_OPEN_META;
case M2RS_CLOSE_REGION:
return ExecutorType.RS_CLOSE_REGION;
case M2RS_CLOSE_ROOT:
return ExecutorType.RS_CLOSE_ROOT;
case M2RS_CLOSE_META:
return ExecutorType.RS_CLOSE_META;
default:
throw new RuntimeException("Unhandled event type " + type);
}
}
/**
* Default constructor.
* @param Name of the hosting server.
*/
public ExecutorService(final String servername) {
super();
this.servername = servername;
}
/**
* Start an executor service with a given name. If there was a service already
* started with the same name, this throws a RuntimeException.
* @param name Name of the service to start.
*/
void startExecutorService(String name, int maxThreads) {
if (this.executorMap.get(name) != null) {
throw new RuntimeException("An executor service with the name " + name +
" is already running!");
}
Executor hbes = new Executor(name, maxThreads, this.eventHandlerListeners);
if (this.executorMap.putIfAbsent(name, hbes) != null) {
throw new RuntimeException("An executor service with the name " + name +
" is already running (2)!");
}
LOG.debug("Starting executor service: " + name);
}
boolean isExecutorServiceRunning(String name) {
return this.executorMap.containsKey(name);
}
public void shutdown() {
for(Entry<String, Executor> entry: this.executorMap.entrySet()) {
List<Runnable> wasRunning =
entry.getValue().threadPoolExecutor.shutdownNow();
if (!wasRunning.isEmpty()) {
LOG.info(entry.getKey() + " had " + wasRunning + " on shutdown");
}
}
this.executorMap.clear();
}
Executor getExecutor(final ExecutorType type) {
return getExecutor(type.getExecutorName(this.servername));
}
Executor getExecutor(String name) {
Executor executor = this.executorMap.get(name);
if (executor == null) {
LOG.debug("Executor service [" + name + "] not found in " + this.executorMap);
}
return executor;
}
public void startExecutorService(final ExecutorType type, final int maxThreads) {
String name = type.getExecutorName(this.servername);
if (isExecutorServiceRunning(name)) {
LOG.debug("Executor service " + toString() + " already running on " +
this.servername);
return;
}
startExecutorService(name, maxThreads);
}
public void submit(final EventHandler eh) {
getExecutor(getExecutorServiceType(eh.getEventType())).submit(eh);
}
/**
* Subscribe to updates before and after processing instances of
* {@link EventHandler.EventType}. Currently only one listener per
* event type.
* @param type Type of event we're registering listener for
* @param listener The listener to run.
* @return The <code>listener</code> that was passed
*/
public void registerListener(final EventHandler.EventType type,
final EventHandlerListener listener) {
this.eventHandlerListeners.put(type, listener);
}
/**
* Stop receiving updates before and after processing instances of
* {@link EventHandler.EventType}
* @param type Type of event we're registering listener for
* @return The listener we removed or null if we did not remove it.
*/
public EventHandlerListener unregisterListener(final EventHandler.EventType type) {
return this.eventHandlerListeners.remove(type);
}
/**
* Executor instance.
*/
private static class Executor {
// default number of threads in the pool
private int corePoolSize = 1;
// how long to retain excess threads
private long keepAliveTimeInMillis = 1000;
// the thread pool executor that services the requests
private final ThreadPoolExecutor threadPoolExecutor;
// work queue to use - unbounded queue
BlockingQueue<Runnable> workQueue = new PriorityBlockingQueue<Runnable>();
private final AtomicInteger threadid = new AtomicInteger(0);
private final String name;
private final Map<EventHandler.EventType, EventHandlerListener> eventHandlerListeners;
protected Executor(String name, int maxThreads,
final Map<EventHandler.EventType, EventHandlerListener> eventHandlerListeners) {
this.name = name;
this.eventHandlerListeners = eventHandlerListeners;
// create the thread pool executor
this.threadPoolExecutor = new ThreadPoolExecutor(corePoolSize, maxThreads,
keepAliveTimeInMillis, TimeUnit.MILLISECONDS, workQueue);
// name the threads for this threadpool
ThreadFactoryBuilder tfb = new ThreadFactoryBuilder();
tfb.setNameFormat(this.name + "-" + this.threadid.incrementAndGet());
this.threadPoolExecutor.setThreadFactory(tfb.build());
}
/**
* Submit the event to the queue for handling.
* @param event
*/
void submit(final EventHandler event) {
// If there is a listener for this type, make sure we call the before
// and after process methods.
EventHandlerListener listener =
this.eventHandlerListeners.get(event.getEventType());
if (listener != null) {
event.setListener(listener);
}
this.threadPoolExecutor.execute(event);
}
}
}

278
src/main/java/org/apache/hadoop/hbase/executor/HBaseEventHandler.java
View File

@ -1,278 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.executor;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.executor.HBaseExecutorService.HBaseExecutorServiceType;
import org.apache.hadoop.hbase.master.ServerManager;
/**
* Abstract base class for all HBase event handlers. Subclasses should
* implement the process() method where the actual handling of the event
* happens.
*
* HBaseEventType is a list of ALL events (which also corresponds to messages -
* either internal to one component or between components). The event type
* names specify the component from which the event originated, and the
* component which is supposed to handle it.
*
* Listeners can listen to all the events by implementing the interface
* HBaseEventHandlerListener, and by registering themselves as a listener. They
* will be called back before and after the process of every event.
*
* TODO: Rename HBaseEvent and HBaseEventType to EventHandler and EventType
* after ZK refactor as it currently would clash with EventType from ZK and
* make the code very confusing.
*/
public abstract class HBaseEventHandler implements Runnable
{
private static final Log LOG = LogFactory.getLog(HBaseEventHandler.class);
// type of event this object represents
protected HBaseEventType eventType = HBaseEventType.NONE;
// is this a region server or master?
protected boolean isRegionServer;
// name of the server - this is needed for naming executors in case of tests
// where region servers may be co-located.
protected String serverName;
// listeners that are called before and after an event is processed
protected static List<HBaseEventHandlerListener> eventHandlerListeners =
Collections.synchronizedList(new ArrayList<HBaseEventHandlerListener>());
/**
* This interface provides hooks to listen to various events received by the
* queue. A class implementing this can listen to the updates by calling
* registerListener and stop receiving updates by calling unregisterListener
*/
public interface HBaseEventHandlerListener {
/**
* Called before any event is processed
*/
public void beforeProcess(HBaseEventHandler event);
/**
* Called after any event is processed
*/
public void afterProcess(HBaseEventHandler event);
}
/**
* These are a list of HBase events that can be handled by the various
* HBaseExecutorService's. All the events are serialized as byte values.
*/
public enum HBaseEventType {
NONE (-1),
// Messages originating from RS (NOTE: there is NO direct communication from
// RS to Master). These are a result of RS updates into ZK.
RS2ZK_REGION_CLOSING (1), // RS is in process of closing a region
RS2ZK_REGION_CLOSED (2), // RS has finished closing a region
RS2ZK_REGION_OPENING (3), // RS is in process of opening a region
RS2ZK_REGION_OPENED (4), // RS has finished opening a region
// Updates from master to ZK. This is done by the master and there is
// nothing to process by either Master or RS
M2ZK_REGION_OFFLINE (50); // Master adds this region as offline in ZK
private final byte value;
/**
* Called by the HMaster. Returns a name of the executor service given an
* event type. Every event type has en entry - if the event should not be
* handled just add the NONE executor.
* @return name of the executor service
*/
public HBaseExecutorServiceType getMasterExecutorForEvent() {
HBaseExecutorServiceType executorServiceType = null;
switch(this) {
case RS2ZK_REGION_CLOSING:
case RS2ZK_REGION_CLOSED:
executorServiceType = HBaseExecutorServiceType.MASTER_CLOSEREGION;
break;
case RS2ZK_REGION_OPENING:
case RS2ZK_REGION_OPENED:
executorServiceType = HBaseExecutorServiceType.MASTER_OPENREGION;
break;
case M2ZK_REGION_OFFLINE:
executorServiceType = HBaseExecutorServiceType.NONE;
break;
default:
throw new RuntimeException("Unhandled event type in the master.");
}
return executorServiceType;
}
/**
* Called by the RegionServer. Returns a name of the executor service given an
* event type. Every event type has en entry - if the event should not be
* handled just return a null executor name.
* @return name of the event service
*/
public static String getRSExecutorForEvent(String serverName) {
throw new RuntimeException("Unsupported operation.");
}
/**
* Start the executor service that handles the passed in event type. The
* server that starts these event executor services wants to handle these
* event types.
*/
public void startMasterExecutorService(String serverName) {
HBaseExecutorServiceType serviceType = getMasterExecutorForEvent();
if(serviceType == HBaseExecutorServiceType.NONE) {
throw new RuntimeException("Event type " + toString() + " not handled on master.");
}
serviceType.startExecutorService(serverName);
}
public static void startRSExecutorService() {
}
HBaseEventType(int intValue) {
this.value = (byte)intValue;
}
public byte getByteValue() {
return value;
}
public static HBaseEventType fromByte(byte value) {
switch(value) {
case -1: return HBaseEventType.NONE;
case 1 : return HBaseEventType.RS2ZK_REGION_CLOSING;
case 2 : return HBaseEventType.RS2ZK_REGION_CLOSED;
case 3 : return HBaseEventType.RS2ZK_REGION_OPENING;
case 4 : return HBaseEventType.RS2ZK_REGION_OPENED;
case 50: return HBaseEventType.M2ZK_REGION_OFFLINE;
default:
throw new RuntimeException("Invalid byte value for conversion to HBaseEventType");
}
}
}
/**
* Default base class constructor.
*
* TODO: isRegionServer and serverName will go away once we do the HMaster
* refactor. We will end up passing a ServerStatus which should tell us both
* the name and if it is a RS or master.
*/
public HBaseEventHandler(boolean isRegionServer, String serverName, HBaseEventType eventType) {
this.isRegionServer = isRegionServer;
this.eventType = eventType;
this.serverName = serverName;
}
/**
* This is a wrapper around process, used to update listeners before and after
* events are processed.
*/
public void run() {
// fire all beforeProcess listeners
for(HBaseEventHandlerListener listener : eventHandlerListeners) {
listener.beforeProcess(this);
}
// call the main process function
try {
process();
} catch(Throwable t) {
LOG.error("Caught throwable while processing event " + eventType, t);
}
// fire all afterProcess listeners
for(HBaseEventHandlerListener listener : eventHandlerListeners) {
LOG.debug("Firing " + listener.getClass().getName() +
".afterProcess event listener for event " + eventType);
listener.afterProcess(this);
}
}
/**
* This method is the main processing loop to be implemented by the various
* subclasses.
*/
public abstract void process();
/**
* Subscribe to updates before and after processing events
*/
public static void registerListener(HBaseEventHandlerListener listener) {
eventHandlerListeners.add(listener);
}
/**
* Stop receiving updates before and after processing events
*/
public static void unregisterListener(HBaseEventHandlerListener listener) {
eventHandlerListeners.remove(listener);
}
public boolean isRegionServer() {
return isRegionServer;
}
/**
* Return the name for this event type.
* @return
*/
public HBaseExecutorServiceType getEventHandlerName() {
// TODO: check for isRegionServer here
return eventType.getMasterExecutorForEvent();
}
/**
* Return the event type
* @return
*/
public HBaseEventType getHBEvent() {
return eventType;
}
/**
* Submits this event object to the correct executor service. This is causes
* this object to get executed by the correct ExecutorService.
*/
public void submit() {
HBaseExecutorServiceType serviceType = getEventHandlerName();
if(serviceType == null) {
throw new RuntimeException("Event " + eventType + " not handled on this server " + serverName);
}
serviceType.getExecutor(serverName).submit(this);
}
/**
* Executes this event object in the caller's thread. This is a synchronous
* way of executing the event.
*/
public void execute() {
this.run();
}
}

171
src/main/java/org/apache/hadoop/hbase/executor/HBaseExecutorService.java
View File

@ -1,171 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.executor;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
* This is a generic HBase executor service. This component abstract a
* threadpool, a queue to which jobs can be submitted and a Runnable that
* handles the object that is added to the queue.
*
* In order to create a new HBExecutorService, you need to do:
* HBExecutorService.startExecutorService("myService");
*
* In order to use the service created above, you need to override the
* HBEventHandler class and create an event type that submits to this service.
*
*/
public class HBaseExecutorService
{
private static final Log LOG = LogFactory.getLog(HBaseExecutorService.class);
// default number of threads in the pool
private int corePoolSize = 1;
// max number of threads - maximum concurrency
private int maximumPoolSize = 5;
// how long to retain excess threads
private long keepAliveTimeInMillis = 1000;
// the thread pool executor that services the requests
ThreadPoolExecutor threadPoolExecutor;
// work queue to use - unbounded queue
BlockingQueue<Runnable> workQueue = new LinkedBlockingQueue<Runnable>();
// name for this executor service
String name;
// hold the all the executors created in a map addressable by their names
static Map<String, HBaseExecutorService> executorServicesMap =
Collections.synchronizedMap(new HashMap<String, HBaseExecutorService>());
/**
* The following is a list of names for the various executor services in both
* the master and the region server.
*/
public enum HBaseExecutorServiceType {
NONE (-1),
MASTER_CLOSEREGION (1),
MASTER_OPENREGION (2);
private final int value;
HBaseExecutorServiceType(int intValue) {
this.value = intValue;
}
public void startExecutorService(String serverName) {
// if this is NONE then there is no executor to start
if(value == NONE.value) {
throw new RuntimeException("Cannot start NONE executor type.");
}
String name = getExecutorName(serverName);
if(HBaseExecutorService.isExecutorServiceRunning(name)) {
LOG.debug("Executor service " + toString() + " already running on " + serverName);
return;
}
HBaseExecutorService.startExecutorService(name);
}
public HBaseExecutorService getExecutor(String serverName) {
// if this is NONE then there is no executor
if(value == NONE.value) {
return null;
}
return HBaseExecutorService.getExecutorService(getExecutorName(serverName));
}
public String getExecutorName(String serverName) {
// if this is NONE then there is no executor
if(value == NONE.value) {
return null;
}
return (this.toString() + "-" + serverName);
}
}
/**
* Start an executor service with a given name. If there was a service already
* started with the same name, this throws a RuntimeException.
* @param name Name of the service to start.
*/
public static void startExecutorService(String name) {
if(executorServicesMap.get(name) != null) {
throw new RuntimeException("An executor service with the name " + name + " is already running!");
}
HBaseExecutorService hbes = new HBaseExecutorService(name);
executorServicesMap.put(name, hbes);
LOG.debug("Starting executor service: " + name);
}
public static boolean isExecutorServiceRunning(String name) {
return (executorServicesMap.containsKey(name));
}
/**
* This method is an accessor for all the HBExecutorServices running so far
* addressable by name. If there is no such service, then it returns null.
*/
public static HBaseExecutorService getExecutorService(String name) {
HBaseExecutorService executor = executorServicesMap.get(name);
if(executor == null) {
LOG.debug("Executor service [" + name + "] not found.");
}
return executor;
}
public static void shutdown() {
for(Entry<String, HBaseExecutorService> entry : executorServicesMap.entrySet()) {
entry.getValue().threadPoolExecutor.shutdown();
}
executorServicesMap.clear();
}
protected HBaseExecutorService(String name) {
this.name = name;
// create the thread pool executor
threadPoolExecutor = new ThreadPoolExecutor(
corePoolSize,
maximumPoolSize,
keepAliveTimeInMillis,
TimeUnit.MILLISECONDS,
workQueue
);
// name the threads for this threadpool
threadPoolExecutor.setThreadFactory(new NamedThreadFactory(name));
}
/**
* Submit the event to the queue for handling.
* @param event
*/
public void submit(Runnable event) {
threadPoolExecutor.execute(event);
}
}

210
src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java Normal file
View File

@ -0,0 +1,210 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.executor;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.hbase.executor.EventHandler.EventType;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Writables;
import org.apache.hadoop.io.Writable;
/**
* Data serialized into ZooKeeper for region transitions.
*/
public class RegionTransitionData implements Writable {
/**
* Type of transition event (offline, opening, opened, closing, closed).
* Required.
*/
private EventType eventType;
/** Region being transitioned. Required. */
private byte [] regionName;
/** Server event originated from. Optional. */
private String serverName;
/** Time the event was created. Required but automatically set. */
private long stamp;
/**
* Writable constructor. Do not use directly.
*/
public RegionTransitionData() {}
/**
* Construct data for a new region transition event with the specified event
* type and region name.
*
* <p>Used when the server name is not known (the master is setting it). This
* happens during cluster startup or during failure scenarios. When
* processing a failed regionserver, the master assigns the regions from that
* server to other servers though the region was never 'closed'. During
* master failover, the new master may have regions stuck in transition
* without a destination so may have to set regions offline and generate a new
* assignment.
*
* <p>Since only the master uses this constructor, the type should always be
* {@link EventType#M2ZK_REGION_OFFLINE}.
*
* @param eventType type of event
* @param regionName name of region
*/
public RegionTransitionData(EventType eventType, byte [] regionName) {
this(eventType, regionName, null);
}
/**
* Construct data for a new region transition event with the specified event
* type, region name, and server name.
*
* <p>Used when the server name is known (a regionserver is setting it).
*
* <p>Valid types for this constructor are {@link EventType#RS2ZK_REGION_CLOSING},
* {@link EventType#RS2ZK_REGION_CLOSED}, {@link EventType#RS2ZK_REGION_OPENING},
* and {@link EventType#RS2ZK_REGION_OPENED}.
*
* @param eventType type of event
* @param regionName name of region
* @param serverName name of server setting data
*/
public RegionTransitionData(EventType eventType, byte [] regionName,
String serverName) {
this.eventType = eventType;
this.stamp = System.currentTimeMillis();
this.regionName = regionName;
this.serverName = serverName;
}
/**
* Gets the type of region transition event.
*
* <p>One of:
* <ul>
* <li>{@link EventType#M2ZK_REGION_OFFLINE}
* <li>{@link EventType#RS2ZK_REGION_CLOSING}
* <li>{@link EventType#RS2ZK_REGION_CLOSED}
* <li>{@link EventType#RS2ZK_REGION_OPENING}
* <li>{@link EventType#RS2ZK_REGION_OPENED}
* </ul>
* @return type of region transition event
*/
public EventType getEventType() {
return eventType;
}
/**
* Gets the name of the region being transitioned.
*
* <p>Region name is required so this never returns null.
* @return region name
*/
public byte [] getRegionName() {
return regionName;
}
/**
* Gets the server the event originated from. If null, this event originated
* from the master.
*
* @return server name of originating regionserver, or null if from master
*/
public String getServerName() {
return serverName;
}
/**
* Gets the timestamp when this event was created.
*
* @return stamp event was created
*/
public long getStamp() {
return stamp;
}
@Override
public void readFields(DataInput in) throws IOException {
// the event type byte
eventType = EventType.values()[in.readShort()];
// the timestamp
stamp = in.readLong();
// the encoded name of the region being transitioned
regionName = Bytes.readByteArray(in);
// remaining fields are optional so prefixed with boolean
// the name of the regionserver sending the data
if(in.readBoolean()) {
serverName = in.readUTF();
} else {
serverName = null;
}
}
@Override
public void write(DataOutput out) throws IOException {
out.writeShort(eventType.ordinal());
out.writeLong(System.currentTimeMillis());
Bytes.writeByteArray(out, regionName);
// remaining fields are optional so prefixed with boolean
out.writeBoolean(serverName != null);
if(serverName != null) {
out.writeUTF(serverName);
}
}
/**
* Get the bytes for this instance. Throws a {@link RuntimeException} if
* there is an error deserializing this instance because it represents a code
* bug.
* @return binary representation of this instance
*/
public byte [] getBytes() {
try {
return Writables.getBytes(this);
} catch(IOException e) {
throw new RuntimeException(e);
}
}
/**
* Get an instance from bytes. Throws a {@link RuntimeException} if
* there is an error serializing this instance from bytes because it
* represents a code bug.
* @param bytes binary representation of this instance
* @return instance of this class
*/
public static RegionTransitionData fromBytes(byte [] bytes) {
try {
RegionTransitionData data = new RegionTransitionData();
Writables.getWritable(bytes, data);
return data;
} catch(IOException e) {
throw new RuntimeException(e);
}
}
@Override
public String toString() {
return "region=" + Bytes.toString(regionName) + ", server=" + serverName +
", state=" + eventType;
}
}

92
src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionEventData.java
View File

@ -1,92 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.executor;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.hbase.HMsg;
import org.apache.hadoop.hbase.executor.HBaseEventHandler.HBaseEventType;
import org.apache.hadoop.io.Writable;
public class RegionTransitionEventData implements Writable {
private HBaseEventType hbEvent;
private String rsName;
private long timeStamp;
private HMsg hmsg;
public RegionTransitionEventData() {
}
public RegionTransitionEventData(HBaseEventType hbEvent, String rsName) {
this(hbEvent, rsName, null);
}
public RegionTransitionEventData(HBaseEventType hbEvent, String rsName, HMsg hmsg) {
this.hbEvent = hbEvent;
this.rsName = rsName;
this.timeStamp = System.currentTimeMillis();
this.hmsg = hmsg;
}
public HBaseEventType getHbEvent() {
return hbEvent;
}
public String getRsName() {
return rsName;
}
public long getTimeStamp() {
return timeStamp;
}
public HMsg getHmsg() {
return hmsg;
}
@Override
public void readFields(DataInput in) throws IOException {
// the event type byte
hbEvent = HBaseEventType.fromByte(in.readByte());
// the hostname of the RS sending the data
rsName = in.readUTF();
// the timestamp
timeStamp = in.readLong();
if(in.readBoolean()) {
// deserialized the HMsg from ZK
hmsg = new HMsg();
hmsg.readFields(in);
}
}
@Override
public void write(DataOutput out) throws IOException {
out.writeByte(hbEvent.getByteValue());
out.writeUTF(rsName);
out.writeLong(System.currentTimeMillis());
out.writeBoolean((hmsg != null));
if(hmsg != null) {
hmsg.write(out);
}
}
}

50
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
View File

@ -23,11 +23,10 @@ import java.io.DataOutput;
import java.io.IOException;
import java.lang.reflect.Array;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.NavigableSet;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@ -45,26 +44,37 @@ import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.MultiAction;
import org.apache.hadoop.hbase.client.Action;
import org.apache.hadoop.hbase.client.MultiResponse;
import org.apache.hadoop.hbase.client.MultiPut;
import org.apache.hadoop.hbase.client.MultiPutResponse;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Row;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.MultiPutResponse;
import org.apache.hadoop.hbase.client.MultiPut;
import org.apache.hadoop.hbase.filter.*;
import org.apache.hadoop.hbase.io.HbaseMapWritable;
import org.apache.hadoop.hbase.filter.BinaryComparator;
import org.apache.hadoop.hbase.filter.ColumnCountGetFilter;
import org.apache.hadoop.hbase.filter.ColumnPrefixFilter;
import org.apache.hadoop.hbase.filter.CompareFilter;
import org.apache.hadoop.hbase.filter.DependentColumnFilter;
import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
import org.apache.hadoop.hbase.filter.InclusiveStopFilter;
import org.apache.hadoop.hbase.filter.PageFilter;
import org.apache.hadoop.hbase.filter.PrefixFilter;
import org.apache.hadoop.hbase.filter.QualifierFilter;
import org.apache.hadoop.hbase.filter.RowFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueExcludeFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
import org.apache.hadoop.hbase.filter.SkipFilter;
import org.apache.hadoop.hbase.filter.ValueFilter;
import org.apache.hadoop.hbase.filter.WhileMatchFilter;
import org.apache.hadoop.hbase.filter.WritableByteArrayComparable;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.wal.HLog;
import org.apache.hadoop.hbase.regionserver.wal.HLogKey;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.io.MapWritable;
import org.apache.hadoop.io.ObjectWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableFactories;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.wal.HLog;
import org.apache.hadoop.hbase.regionserver.wal.HLogKey;
import org.apache.hadoop.hbase.util.Bytes;
/**
* This is a customized version of the polymorphic hadoop
@ -173,16 +183,10 @@ public class HbaseObjectWritable implements Writable, Configurable {
addToMap(HLog.Entry[].class, code++);
addToMap(HLogKey.class, code++);
// List
addToMap(List.class, code++);
addToMap(NavigableSet.class, code++);
addToMap(ColumnPrefixFilter.class, code++);
// Multi
addToMap(Row.class, code++);
addToMap(Action.class, code++);
addToMap(MultiAction.class, code++);
addToMap(MultiResponse.class, code++);
}
private Class<?> declaredClass;
@ -512,4 +516,4 @@ public class HbaseObjectWritable implements Writable, Configurable {
public Configuration getConf() {
return this.conf;
}
}
}

5
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
View File

@ -345,8 +345,9 @@ public class HBaseRPC {
if (maxAttempts >= 0 && ++reconnectAttempts >= maxAttempts) {
LOG.info("Server at " + addr + " could not be reached after " +
reconnectAttempts + " tries, giving up.");
throw new RetriesExhaustedException("Failed setting up proxy to " +
addr.toString() + " after attempts=" + reconnectAttempts);
throw new RetriesExhaustedException("Failed setting up proxy " +
protocol + " to " + addr.toString() + " after attempts=" +
reconnectAttempts, se);
}
} catch(SocketTimeoutException te) { // namenode is busy
LOG.info("Problem connecting to server: " + addr);

3
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPCProtocolVersion.java
View File

@ -76,7 +76,8 @@ public interface HBaseRPCProtocolVersion extends VersionedProtocol {
* <li>Version 22: HBASE-2209. Added List support to RPC</li>
* <li>Version 23: HBASE-2066, multi-put.</li>
* <li>Version 24: HBASE-2473, create table with regions.</li>
* <li>Version 25: Added openRegion and Stoppable/Abortable to API.</li>
* </ul>
*/
public static final long versionID = 24L;
public static final long versionID = 25L;
}

44
src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
View File

@ -19,13 +19,12 @@
*/
package org.apache.hadoop.hbase.ipc;
import java.io.IOException;
import org.apache.hadoop.hbase.ClusterStatus;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.io.Writable;
import java.io.IOException;
import org.apache.hadoop.hbase.UnknownRegionException;
/**
* Clients interact with the HMasterInterface to gain access to meta-level
@ -73,12 +72,10 @@ public interface HMasterInterface extends HBaseRPCProtocolVersion {
/**
* Modifies an existing column on the specified table
* @param tableName table name
* @param columnName name of the column to edit
* @param descriptor new column descriptor
* @throws IOException e
*/
public void modifyColumn(final byte [] tableName, final byte [] columnName,
HColumnDescriptor descriptor)
public void modifyColumn(final byte [] tableName, HColumnDescriptor descriptor)
throws IOException;
@ -110,12 +107,11 @@ public interface HMasterInterface extends HBaseRPCProtocolVersion {
* Modify a table's metadata
*
* @param tableName table to modify
* @param op the operation to do
* @param args arguments for operation
* @param htd new descriptor for table
* @throws IOException e
*/
public void modifyTable(byte[] tableName, HConstants.Modify op, Writable[] args)
throws IOException;
public void modifyTable(byte[] tableName, HTableDescriptor htd)
throws IOException;
/**
* Shutdown an HBase cluster.
@ -123,9 +119,33 @@ public interface HMasterInterface extends HBaseRPCProtocolVersion {
*/
public void shutdown() throws IOException;
/**
* Stop HBase Master only.
* Does not shutdown the cluster.
* @throws IOException e
*/
public void stopMaster() throws IOException;
/**
* Return cluster status.
* @return status object
*/
public ClusterStatus getClusterStatus();
}
/**
* Move the region <code>r</code> to <code>dest</code>.
* @param encodedRegionName The encoded region name.
* @param destServerName The servername of the destination regionserver
* @throws UnknownRegionException Thrown if we can't find a region named
* <code>encodedRegionName</code>
*/
public void move(final byte [] encodedRegionName, final byte [] destServerName)
throws UnknownRegionException;
/**
* @param b If true, enable balancer. If false, disable balancer.
* @return Previous balancer value
*/
public boolean balance(final boolean b);
}

87
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
View File

@ -19,31 +19,31 @@
*/
package org.apache.hadoop.hbase.ipc;
import java.io.IOException;
import java.util.List;
import java.util.NavigableSet;
import org.apache.hadoop.hbase.Abortable;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.NotServingRegionException;
import org.apache.hadoop.hbase.Stoppable;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.MultiAction;
import org.apache.hadoop.hbase.client.MultiResponse;
import org.apache.hadoop.hbase.client.MultiPut;
import org.apache.hadoop.hbase.client.MultiPutResponse;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.wal.HLog;
import java.io.IOException;
import java.util.List;
/**
* Clients interact with HRegionServers using a handle to the HRegionInterface.
*
* <p>NOTE: if you change the interface, you must change the RPC version
* number in HBaseRPCProtocolVersion
*/
public interface HRegionInterface extends HBaseRPCProtocolVersion {
public interface HRegionInterface extends HBaseRPCProtocolVersion, Stoppable, Abortable {
/**
* Get metainfo about an HRegion
*
@ -69,12 +69,6 @@ public interface HRegionInterface extends HBaseRPCProtocolVersion {
final byte [] row, final byte [] family)
throws IOException;
/**
*
* @return the regions served by this regionserver
*/
public HRegion [] getOnlineRegionsAsArray();
/**
* Perform Get operation.
* @param regionName name of region to get from
@ -260,11 +254,10 @@ public interface HRegionInterface extends HBaseRPCProtocolVersion {
/**
* Method used when a master is taking the place of another failed one.
* @return All regions assigned on this region server
* @return All regions online on this region server
* @throws IOException e
*/
public HRegionInfo[] getRegionsAssignment() throws IOException;
public NavigableSet<HRegionInfo> getOnlineRegions();
/**
* Method used when a master is taking the place of another failed one.
@ -273,13 +266,6 @@ public interface HRegionInterface extends HBaseRPCProtocolVersion {
*/
public HServerInfo getHServerInfo() throws IOException;
/**
* Method used for doing multiple actions(Deletes, Gets and Puts) in one call
* @param multi
* @return MultiResult
* @throws IOException
*/
public MultiResponse multi(MultiAction multi) throws IOException;
/**
* Multi put for putting multiple regions worth of puts at once.
@ -296,6 +282,60 @@ public interface HRegionInterface extends HBaseRPCProtocolVersion {
public void bulkLoadHFile(String hfilePath,
byte[] regionName, byte[] familyName) throws IOException;
// Master methods
/**
* Opens the specified region.
* @param region region to open
*/
public void openRegion(final HRegionInfo region);
/**
* Closes the specified region.
* @param region region to close
* @return true if closing region, false if not
*/
public boolean closeRegion(final HRegionInfo region)
throws NotServingRegionException;
// Region administrative methods
/**
* Flushes the MemStore of the specified region.
* <p>
* This method is synchronous.
* @param regionInfo region to flush
* @throws NotServingRegionException
* @throws IOException
*/
void flushRegion(HRegionInfo regionInfo)
throws NotServingRegionException, IOException;
/**
* Splits the specified region.
* <p>
* This method currently flushes the region and then forces a compaction which
* will then trigger a split. The flush is done synchronously but the
* compaction is asynchronous.
* @param regionInfo region to split
* @throws NotServingRegionException
* @throws IOException
*/
void splitRegion(HRegionInfo regionInfo)
throws NotServingRegionException, IOException;
/**
* Compacts the specified region. Performs a major compaction if specified.
* <p>
* This method is asynchronous.
* @param regionInfo region to compact
* @param major true to force major compaction
* @throws NotServingRegionException
* @throws IOException
*/
void compactRegion(HRegionInfo regionInfo, boolean major)
throws NotServingRegionException, IOException;
/**
* Replicates the given entries. The guarantee is that the given entries
* will be durable on the slave cluster if this method returns without
@ -306,5 +346,4 @@ public interface HRegionInterface extends HBaseRPCProtocolVersion {
* @throws IOException
*/
public void replicateLogEntries(HLog.Entry[] entries) throws IOException;
}

173
src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java Normal file
View File

@ -0,0 +1,173 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.zookeeper.ZKUtil;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperListener;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher;
import org.apache.zookeeper.KeeperException;
/**
* Handles everything on master-side related to master election.
*
* <p>Listens and responds to ZooKeeper notifications on the master znode,
* both <code>nodeCreated</code> and <code>nodeDeleted</code>.
*
* <p>Contains blocking methods which will hold up backup masters, waiting
* for the active master to fail.
*
* <p>This class is instantiated in the HMaster constructor and the method
* {@link #blockUntilBecomingActiveMaster()} is called to wait until becoming
* the active master of the cluster.
*/
class ActiveMasterManager extends ZooKeeperListener {
private static final Log LOG = LogFactory.getLog(ActiveMasterManager.class);
final AtomicBoolean clusterHasActiveMaster = new AtomicBoolean(false);
private final HServerAddress address;
private final Server master;
ActiveMasterManager(ZooKeeperWatcher watcher, HServerAddress address,
Server master) {
super(watcher);
this.address = address;
this.master = master;
}
@Override
public void nodeCreated(String path) {
if(path.equals(watcher.masterAddressZNode) && !master.isStopped()) {
handleMasterNodeChange();
}
}
@Override
public void nodeDeleted(String path) {
if(path.equals(watcher.masterAddressZNode) && !master.isStopped()) {
handleMasterNodeChange();
}
}
/**
* Handle a change in the master node. Doesn't matter whether this was called
* from a nodeCreated or nodeDeleted event because there are no guarantees
* that the current state of the master node matches the event at the time of
* our next ZK request.
*
* <p>Uses the watchAndCheckExists method which watches the master address node
* regardless of whether it exists or not. If it does exist (there is an
* active master), it returns true. Otherwise it returns false.
*
* <p>A watcher is set which guarantees that this method will get called again if
* there is another change in the master node.
*/
private void handleMasterNodeChange() {
// Watch the node and check if it exists.
try {
synchronized(clusterHasActiveMaster) {
if(ZKUtil.watchAndCheckExists(watcher, watcher.masterAddressZNode)) {
// A master node exists, there is an active master
LOG.debug("A master is now available");
clusterHasActiveMaster.set(true);
} else {
// Node is no longer there, cluster does not have an active master
LOG.debug("No master available. notifying waiting threads");
clusterHasActiveMaster.set(false);
// Notify any thread waiting to become the active master
clusterHasActiveMaster.notifyAll();
}
}
} catch (KeeperException ke) {
master.abort("Received an unexpected KeeperException, aborting", ke);
}
}
/**
* Block until becoming the active master.
*
* Method blocks until there is not another active master and our attempt
* to become the new active master is successful.
*
* This also makes sure that we are watching the master znode so will be
* notified if another master dies.
* @return False if we did not start up this cluster, another
* master did, or if a problem (zookeeper, stop flag has been set on this
* Master)
*/
boolean blockUntilBecomingActiveMaster() {
boolean thisMasterStartedCluster = true;
// Try to become the active master, watch if there is another master
try {
if(ZKUtil.setAddressAndWatch(watcher, watcher.masterAddressZNode,
address)) {
// We are the master, return
clusterHasActiveMaster.set(true);
return thisMasterStartedCluster;
}
} catch (KeeperException ke) {
master.abort("Received an unexpected KeeperException, aborting", ke);
return false;
}
// There is another active master, this is not a cluster startup
// and we must wait until the active master dies
LOG.info("Another master is already the active master, waiting to become " +
"the next active master");
clusterHasActiveMaster.set(true);
thisMasterStartedCluster = false;
synchronized(clusterHasActiveMaster) {
while(clusterHasActiveMaster.get() && !master.isStopped()) {
try {
clusterHasActiveMaster.wait();
} catch (InterruptedException e) {
// We expect to be interrupted when a master dies, will fall out if so
LOG.debug("Interrupted waiting for master to die", e);
}
}
if(master.isStopped()) {
return thisMasterStartedCluster;
}
// Try to become active master again now that there is no active master
blockUntilBecomingActiveMaster();
}
return thisMasterStartedCluster;
}
public void stop() {
try {
// If our address is in ZK, delete it on our way out
HServerAddress zkAddress =
ZKUtil.getDataAsAddress(watcher, watcher.masterAddressZNode);
// TODO: redo this to make it atomic (only added for tests)
if(zkAddress != null &&
zkAddress.equals(address)) {
ZKUtil.deleteNode(watcher, watcher.masterAddressZNode);
}
} catch (KeeperException e) {
watcher.error("Error deleting our own master address node", e);
}
}
}

1188
src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java Normal file
View File

File diff suppressed because it is too large Load Diff

620
src/main/java/org/apache/hadoop/hbase/master/BaseScanner.java
View File

@ -1,620 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.NavigableMap;
import java.util.TreeMap;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathFilter;
import org.apache.hadoop.hbase.Chore;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.UnknownScannerException;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.Store;
import org.apache.hadoop.hbase.regionserver.StoreFile;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Writables;
import org.apache.hadoop.io.BooleanWritable;
import org.apache.hadoop.ipc.RemoteException;
/**
* Base HRegion scanner class. Holds utilty common to <code>ROOT</code> and
* <code>META</code> HRegion scanners.
*
* <p>How do we know if all regions are assigned? After the initial scan of
* the <code>ROOT</code> and <code>META</code> regions, all regions known at
* that time will have been or are in the process of being assigned.</p>
*
* <p>When a region is split the region server notifies the master of the
* split and the new regions are assigned. But suppose the master loses the
* split message? We need to periodically rescan the <code>ROOT</code> and
* <code>META</code> regions.
* <ul>
* <li>If we rescan, any regions that are new but not assigned will have
* no server info. Any regions that are not being served by the same
* server will get re-assigned.</li>
*
* <li>Thus a periodic rescan of the root region will find any new
* <code>META</code> regions where we missed the <code>META</code> split
* message or we failed to detect a server death and consequently need to
* assign the region to a new server.</li>
*
* <li>if we keep track of all the known <code>META</code> regions, then
* we can rescan them periodically. If we do this then we can detect any
* regions for which we missed a region split message.</li>
* </ul>
*
* Thus just keeping track of all the <code>META</code> regions permits
* periodic rescanning which will detect unassigned regions (new or
* otherwise) without the need to keep track of every region.</p>
*
* <p>So the <code>ROOT</code> region scanner needs to wake up:
* <ol>
* <li>when the master receives notification that the <code>ROOT</code>
* region has been opened.</li>
* <li>periodically after the first scan</li>
* </ol>
*
* The <code>META</code> scanner needs to wake up:
* <ol>
* <li>when a <code>META</code> region comes on line</li>
* </li>periodically to rescan the online <code>META</code> regions</li>
* </ol>
*
* <p>A <code>META</code> region is not 'online' until it has been scanned
* once.
*/
abstract class BaseScanner extends Chore {
static final Log LOG = LogFactory.getLog(BaseScanner.class.getName());
// These are names of new columns in a meta region offlined parent row. They
// are added by the metascanner after we verify that split daughter made it
// in. Their value is 'true' if present.
private static final byte[] SPLITA_CHECKED =
Bytes.toBytes(Bytes.toString(HConstants.SPLITA_QUALIFIER) + "_checked");
private static final byte[] SPLITB_CHECKED =
Bytes.toBytes(Bytes.toString(HConstants.SPLITB_QUALIFIER) + "_checked");
// Make the 'true' Writable once only.
private static byte[] TRUE_WRITABLE_AS_BYTES;
static {
try {
TRUE_WRITABLE_AS_BYTES = Writables.getBytes(new BooleanWritable(true));
} catch (IOException e) {
e.printStackTrace();
}
}
private final boolean rootRegion;
protected final HMaster master;
protected boolean initialScanComplete;
protected abstract boolean initialScan();
protected abstract void maintenanceScan();
// will use this variable to synchronize and make sure we aren't interrupted
// mid-scan
final Object scannerLock = new Object();
BaseScanner(final HMaster master, final boolean rootRegion,
final AtomicBoolean stop) {
super("Scanner for " + (rootRegion ? "-ROOT-":".META.") + " table",
master.getConfiguration().
getInt("hbase.master.meta.thread.rescanfrequency", 60 * 1000), stop);
this.rootRegion = rootRegion;
this.master = master;
this.initialScanComplete = false;
}
/** @return true if initial scan completed successfully */
public boolean isInitialScanComplete() {
return initialScanComplete;
}
@Override
protected boolean initialChore() {
return initialScan();
}
@Override
protected void chore() {
maintenanceScan();
}
/**
* @param region Region to scan
* @throws IOException
*/
protected void scanRegion(final MetaRegion region) throws IOException {
HRegionInterface regionServer = null;
long scannerId = -1L;
LOG.info(Thread.currentThread().getName() + " scanning meta region " +
region.toString());
// Array to hold list of split parents found. Scan adds to list. After
// scan we go check if parents can be removed and that their daughters
// are in place.
NavigableMap<HRegionInfo, Result> splitParents =
new TreeMap<HRegionInfo, Result>();
List<byte []> emptyRows = new ArrayList<byte []>();
int rows = 0;
try {
regionServer =
this.master.getServerConnection().getHRegionConnection(region.getServer());
Scan s = new Scan().addFamily(HConstants.CATALOG_FAMILY);
// Make this scan do a row at a time otherwise, data can be stale.
s.setCaching(1);
scannerId = regionServer.openScanner(region.getRegionName(), s);
while (true) {
Result values = regionServer.next(scannerId);
if (values == null || values.size() == 0) {
break;
}
HRegionInfo info = master.getHRegionInfo(values.getRow(), values);
if (info == null) {
emptyRows.add(values.getRow());
continue;
}
String serverAddress = getServerAddress(values);
long startCode = getStartCode(values);
// Note Region has been assigned.
checkAssigned(regionServer, region, info, serverAddress, startCode, true);
if (isSplitParent(info)) {
splitParents.put(info, values);
}
rows += 1;
}
if (rootRegion) {
this.master.getRegionManager().setNumMetaRegions(rows);
}
} catch (IOException e) {
if (e instanceof RemoteException) {
e = RemoteExceptionHandler.decodeRemoteException((RemoteException) e);
if (e instanceof UnknownScannerException) {
// Reset scannerId so we do not try closing a scanner the other side
// has lost account of: prevents duplicated stack trace out of the
// below close in the finally.
scannerId = -1L;
}
}
throw e;
} finally {
try {
if (scannerId != -1L && regionServer != null) {
regionServer.close(scannerId);
}
} catch (IOException e) {
LOG.error("Closing scanner",
RemoteExceptionHandler.checkIOException(e));
}
}
// Scan is finished.
// First clean up any meta region rows which had null HRegionInfos
if (emptyRows.size() > 0) {
LOG.warn("Found " + emptyRows.size() + " rows with empty HRegionInfo " +
"while scanning meta region " + Bytes.toString(region.getRegionName()));
this.master.deleteEmptyMetaRows(regionServer, region.getRegionName(),
emptyRows);
}
// Take a look at split parents to see if any we can clean up any and to
// make sure that daughter regions are in place.
if (splitParents.size() > 0) {
for (Map.Entry<HRegionInfo, Result> e : splitParents.entrySet()) {
HRegionInfo hri = e.getKey();
cleanupAndVerifySplits(region.getRegionName(), regionServer,
hri, e.getValue());
}
}
LOG.info(Thread.currentThread().getName() + " scan of " + rows +
" row(s) of meta region " + region.toString() + " complete");
}
/*
* @param r
* @return Empty String or server address found in <code>r</code>
*/
static String getServerAddress(final Result r) {
final byte[] val = r.getValue(HConstants.CATALOG_FAMILY,
HConstants.SERVER_QUALIFIER);
return val == null || val.length <= 0 ? "" : Bytes.toString(val);
}
/*
* @param r
* @return Return 0L or server startcode found in <code>r</code>
*/
static long getStartCode(final Result r) {
final byte[] val = r.getValue(HConstants.CATALOG_FAMILY,
HConstants.STARTCODE_QUALIFIER);
return val == null || val.length <= 0 ? 0L : Bytes.toLong(val);
}
/*
* @param info Region to check.
* @return True if this is a split parent.
*/
private boolean isSplitParent(final HRegionInfo info) {
if (!info.isSplit()) {
return false;
}
if (!info.isOffline()) {
LOG.warn("Region is split but not offline: " +
info.getRegionNameAsString());
}
return true;
}
/*
* If daughters no longer hold reference to the parents, delete the parent.
* If the parent is lone without daughter splits AND there are references in
* the filesystem, then a daughters was not added to .META. -- must have been
* a crash before their addition. Add them here.
* @param metaRegionName Meta region name: e.g. .META.,,1
* @param server HRegionInterface of meta server to talk to
* @param parent HRegionInfo of split offlined parent
* @param rowContent Content of <code>parent</code> row in
* <code>metaRegionName</code>
* @return True if we removed <code>parent</code> from meta table and from
* the filesystem.
* @throws IOException
*/
private boolean cleanupAndVerifySplits(final byte [] metaRegionName,
final HRegionInterface srvr, final HRegionInfo parent,
Result rowContent)
throws IOException {
boolean result = false;
// Run checks on each daughter split.
boolean hasReferencesA = checkDaughter(metaRegionName, srvr,
parent, rowContent, HConstants.SPLITA_QUALIFIER);
boolean hasReferencesB = checkDaughter(metaRegionName, srvr,
parent, rowContent, HConstants.SPLITB_QUALIFIER);
if (!hasReferencesA && !hasReferencesB) {
LOG.info("Deleting region " + parent.getRegionNameAsString() +
" because daughter splits no longer hold references");
HRegion.deleteRegion(this.master.getFileSystem(),
this.master.getRootDir(), parent);
HRegion.removeRegionFromMETA(srvr, metaRegionName,
parent.getRegionName());
result = true;
}
return result;
}
/*
* See if the passed daughter has references in the filesystem to the parent
* and if not, remove the note of daughter region in the parent row: its
* column info:splitA or info:splitB. Also make sure that daughter row is
* present in the .META. and mark the parent row when confirmed so we don't
* keep checking. The mark will be info:splitA_checked and its value will be
* a true BooleanWritable.
* @param metaRegionName
* @param srvr
* @param parent
* @param rowContent
* @param qualifier
* @return True if this daughter still has references to the parent.
* @throws IOException
*/
private boolean checkDaughter(final byte [] metaRegionName,
final HRegionInterface srvr, final HRegionInfo parent,
final Result rowContent, final byte [] qualifier)
throws IOException {
HRegionInfo hri = getDaughterRegionInfo(rowContent, qualifier);
boolean references = hasReferences(metaRegionName, srvr, parent, rowContent,
hri, qualifier);
// Return if no references.
if (!references) return references;
if (!verifyDaughterRowPresent(rowContent, qualifier, srvr, metaRegionName,
hri, parent)) {
// If we got here, then the parent row does not yet have the
// "daughter row verified present" marker present. Add it.
addDaughterRowChecked(metaRegionName, srvr, parent.getRegionName(), hri,
qualifier);
}
return references;
}
/*
* Check the daughter of parent is present in meta table. If not there,
* add it.
* @param rowContent
* @param daughter
* @param srvr
* @param metaRegionName
* @param daughterHRI
* @throws IOException
* @return True, if parent row has marker for "daughter row verified present"
* else, false (and will do fixup adding daughter if daughter not present).
*/
private boolean verifyDaughterRowPresent(final Result rowContent,
final byte [] daughter, final HRegionInterface srvr,
final byte [] metaRegionName,
final HRegionInfo daughterHRI, final HRegionInfo parent)
throws IOException {
// See if the 'checked' column is in parent. If so, we're done.
boolean present = getDaughterRowChecked(rowContent, daughter);
if (present) return present;
// Parent is not carrying the splitA_checked/splitB_checked so this must
// be the first time through here checking splitA/splitB are in metatable.
byte [] daughterRowKey = daughterHRI.getRegionName();
Get g = new Get(daughterRowKey);
g.addColumn(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER);
Result r = srvr.get(metaRegionName, g);
if (r == null || r.isEmpty()) {
// Daughter row not present. Fixup kicks in. Insert it.
LOG.warn("Fixup broke split: Add missing split daughter to meta," +
" daughter=" + daughterHRI.toString() + ", parent=" + parent.toString());
Put p = new Put(daughterRowKey);
p.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER,
Writables.getBytes(daughterHRI));
srvr.put(metaRegionName, p);
}
return present;
}
/*
* Add to parent a marker that we verified the daughter exists.
* @param metaRegionName
* @param srvr
* @param parent
* @param split
* @param daughter
* @throws IOException
*/
private void addDaughterRowChecked(final byte [] metaRegionName,
final HRegionInterface srvr, final byte [] parent,
final HRegionInfo split, final byte [] daughter)
throws IOException {
Put p = new Put(parent);
p.add(HConstants.CATALOG_FAMILY, getNameOfVerifiedDaughterColumn(daughter),
TRUE_WRITABLE_AS_BYTES);
srvr.put(metaRegionName, p);
}
/*
* @param rowContent
* @param which
* @return True if the daughter row has already been verified present in
* metatable.
* @throws IOException
*/
private boolean getDaughterRowChecked(final Result rowContent,
final byte[] which)
throws IOException {
final byte[] b = rowContent.getValue(HConstants.CATALOG_FAMILY,
getNameOfVerifiedDaughterColumn(which));
BooleanWritable bw = null;
if (b != null && b.length > 0) {
bw = (BooleanWritable)Writables.getWritable(b, new BooleanWritable());
}
return bw == null? false: bw.get();
}
/*
* @param daughter
* @return Returns splitA_checked or splitB_checked dependent on what
* <code>daughter</code> is.
*/
private static byte [] getNameOfVerifiedDaughterColumn(final byte [] daughter) {
return (Bytes.equals(HConstants.SPLITA_QUALIFIER, daughter)
? SPLITA_CHECKED : SPLITB_CHECKED);
}
/*
* Get daughter HRegionInfo out of parent info:splitA/info:splitB columns.
* @param rowContent
* @param which Whether "info:splitA" or "info:splitB" column
* @return Deserialized content of the info:splitA or info:splitB as a
* HRegionInfo
* @throws IOException
*/
private HRegionInfo getDaughterRegionInfo(final Result rowContent,
final byte [] which)
throws IOException {
return Writables.getHRegionInfoOrNull(
rowContent.getValue(HConstants.CATALOG_FAMILY, which));
}
/*
* Remove mention of daughter from parent row.
* parent row.
* @param metaRegionName
* @param srvr
* @param parent
* @param split
* @param qualifier
* @throws IOException
*/
private void removeDaughterFromParent(final byte [] metaRegionName,
final HRegionInterface srvr, final HRegionInfo parent,
final HRegionInfo split, final byte [] qualifier)
throws IOException {
if (LOG.isDebugEnabled()) {
LOG.debug(split.getRegionNameAsString() +
" no longer has references to " + parent.getRegionNameAsString());
}
Delete delete = new Delete(parent.getRegionName());
delete.deleteColumns(HConstants.CATALOG_FAMILY, qualifier);
srvr.delete(metaRegionName, delete);
}
/*
* Checks if a daughter region -- either splitA or splitB -- still holds
* references to parent. If not, removes reference to the split from
* the parent meta region row so we don't check it any more.
* @param metaRegionName Name of meta region to look in.
* @param srvr Where region resides.
* @param parent Parent region name.
* @param rowContent Keyed content of the parent row in meta region.
* @param split Which column family.
* @param qualifier Which of the daughters to look at, splitA or splitB.
* @return True if still has references to parent.
* @throws IOException
*/
private boolean hasReferences(final byte [] metaRegionName,
final HRegionInterface srvr, final HRegionInfo parent,
Result rowContent, final HRegionInfo split, byte [] qualifier)
throws IOException {
boolean result = false;
if (split == null) {
return result;
}
Path tabledir =
new Path(this.master.getRootDir(), split.getTableDesc().getNameAsString());
for (HColumnDescriptor family: split.getTableDesc().getFamilies()) {
Path p = Store.getStoreHomedir(tabledir, split.getEncodedName(),
family.getName());
if (!this.master.getFileSystem().exists(p)) continue;
// Look for reference files. Call listStatus with an anonymous
// instance of PathFilter.
FileStatus [] ps =
this.master.getFileSystem().listStatus(p, new PathFilter () {
public boolean accept(Path path) {
return StoreFile.isReference(path);
}
}
);
if (ps != null && ps.length > 0) {
result = true;
break;
}
}
if (!result) {
removeDaughterFromParent(metaRegionName, srvr, parent, split, qualifier);
}
return result;
}
/*
* Check the passed region is assigned. If not, add to unassigned.
* @param regionServer
* @param meta
* @param info
* @param hostnameAndPort hostname ':' port as it comes out of .META.
* @param startCode
* @param checkTwice should we check twice before adding a region
* to unassigned pool.
* @throws IOException
*/
protected void checkAssigned(final HRegionInterface regionServer,
final MetaRegion meta, HRegionInfo info,
final String hostnameAndPort, final long startCode, boolean checkTwice)
throws IOException {
boolean tryAgain = false;
String serverName = null;
String sa = hostnameAndPort;
long sc = startCode;
if (sa == null || sa.length() <= 0) {
// Scans are sloppy. They cache a row internally so may have data that
// is a little stale. Make sure that for sure this serverAddress is null.
// We are trying to avoid double-assignments. See hbase-1784.
Get g = new Get(info.getRegionName());
g.addFamily(HConstants.CATALOG_FAMILY);
Result r = regionServer.get(meta.getRegionName(), g);
if (r != null && !r.isEmpty()) {
sa = getServerAddress(r);
sc = getStartCode(r);
info = master.getHRegionInfo(r.getRow(), r);
}
}
if (sa != null && sa.length() > 0) {
serverName = HServerInfo.getServerName(sa, sc);
}
HServerInfo storedInfo = null;
synchronized (this.master.getRegionManager()) {
/* We don't assign regions that are offline, in transition or were on
* a dead server. Regions that were on a dead server will get reassigned
* by ProcessServerShutdown
*/
if (info == null || info.isOffline() ||
this.master.getRegionManager().regionIsInTransition(info.getRegionNameAsString()) ||
(serverName != null && this.master.getServerManager().isDead(serverName))) {
return;
}
if (serverName != null) {
storedInfo = this.master.getServerManager().getServerInfo(serverName);
}
// If we can't find the HServerInfo, then add it to the list of
// unassigned regions.
if (storedInfo == null) {
if (checkTwice) {
tryAgain = true;
} else {
if (LOG.isDebugEnabled()) {
LOG.debug("Current assignment of " + info.getRegionNameAsString() +
" is not valid; " + " serverAddress=" + sa +
", startCode=" + sc + " unknown.");
}
// Now get the region assigned
this.master.getRegionManager().setUnassigned(info, true);
}
}
}
if (tryAgain) {
// The current assignment is invalid. But we need to try again.
if (LOG.isDebugEnabled()) {
LOG.debug("Current assignment of " + info.getRegionNameAsString() +
" is not valid; " + " serverAddress=" + sa +
", startCode=" + sc + " unknown; checking once more!");
}
// passing null for hostNameAndPort will force the function
// to reget the assignment from META and protect against
// double assignment race conditions (HBASE-2755).
checkAssigned(regionServer, meta, info, null, 0, false);
}
}
/**
* Interrupt thread regardless of what it's doing
*/
public void interruptAndStop() {
synchronized(scannerLock){
if (isAlive()) {
super.interrupt();
LOG.info("Interrupted");
}
}
}
}

162
src/main/java/org/apache/hadoop/hbase/master/ChangeTableState.java
View File

@ -1,162 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Writables;
import java.io.IOException;
import java.util.HashSet;
import java.util.Map;
import java.util.TreeMap;
/**
* Instantiated to enable or disable a table
*/
class ChangeTableState extends TableOperation {
private final Log LOG = LogFactory.getLog(this.getClass());
private boolean online;
// Do in order.
protected final Map<String, HashSet<HRegionInfo>> servedRegions =
new TreeMap<String, HashSet<HRegionInfo>>();
protected long lockid;
ChangeTableState(final HMaster master, final byte [] tableName,
final boolean onLine)
throws IOException {
super(master, tableName);
this.online = onLine;
}
@Override
protected void processScanItem(String serverName, HRegionInfo info) {
if (isBeingServed(serverName)) {
HashSet<HRegionInfo> regions = this.servedRegions.get(serverName);
if (regions == null) {
regions = new HashSet<HRegionInfo>();
}
regions.add(info);
this.servedRegions.put(serverName, regions);
}
}
@Override
protected void postProcessMeta(MetaRegion m, HRegionInterface server)
throws IOException {
// Process regions not being served
if (LOG.isDebugEnabled()) {
LOG.debug("Processing unserved regions");
}
for (HRegionInfo i: this.unservedRegions) {
if (i.isOffline() && i.isSplit()) {
if (LOG.isDebugEnabled()) {
LOG.debug("Skipping region " + i.toString() +
" because it is offline and split");
}
continue;
}
if(!this.online && this.master.getRegionManager().
isPendingOpen(i.getRegionNameAsString())) {
LOG.debug("Skipping region " + i.toString() +
" because it is pending open, will tell it to close later");
continue;
}
// If it's already offline then don't set it a second/third time, skip
// Same for online, don't set again if already online
if (!(i.isOffline() && !online) && !(!i.isOffline() && online)) {
// Update meta table
Put put = updateRegionInfo(i);
server.put(m.getRegionName(), put);
Delete delete = new Delete(i.getRegionName());
delete.deleteColumns(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER);
delete.deleteColumns(HConstants.CATALOG_FAMILY, HConstants.STARTCODE_QUALIFIER);
server.delete(m.getRegionName(), delete);
}
if (LOG.isDebugEnabled()) {
LOG.debug("Removed server and startcode from row and set online=" +
this.online + ": " + i.getRegionNameAsString());
}
synchronized (master.getRegionManager()) {
if (this.online) {
// Bring offline regions on-line
if (!this.master.getRegionManager().regionIsOpening(i.getRegionNameAsString())) {
this.master.getRegionManager().setUnassigned(i, false);
}
} else {
// Prevent region from getting assigned.
this.master.getRegionManager().removeRegion(i);
}
}
}
// Process regions currently being served
if (LOG.isDebugEnabled()) {
LOG.debug("Processing regions currently being served");
}
synchronized (this.master.getRegionManager()) {
for (Map.Entry<String, HashSet<HRegionInfo>> e:
this.servedRegions.entrySet()) {
String serverName = e.getKey();
if (this.online) {
LOG.debug("Already online");
continue; // Already being served
}
// Cause regions being served to be taken off-line and disabled
for (HRegionInfo i: e.getValue()) {
// The scan we did could be totally staled, get the freshest data
Get get = new Get(i.getRegionName());
get.addColumn(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER);
Result values = server.get(m.getRegionName(), get);
String serverAddress = BaseScanner.getServerAddress(values);
// If this region is unassigned, skip!
if(serverAddress.length() == 0) {
continue;
}
if (LOG.isDebugEnabled()) {
LOG.debug("Adding region " + i.getRegionNameAsString() +
" to setClosing list");
}
// this marks the regions to be closed
this.master.getRegionManager().setClosing(serverName, i, true);
}
}
}
this.servedRegions.clear();
}
protected Put updateRegionInfo(final HRegionInfo i)
throws IOException {
i.setOffline(!online);
Put put = new Put(i.getRegionName());
put.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER, Writables.getBytes(i));
return put;
}
}

58
src/main/java/org/apache/hadoop/hbase/master/ColumnOperation.java
View File

@ -1,58 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.TableNotDisabledException;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Writables;
import java.io.IOException;
abstract class ColumnOperation extends TableOperation {
private final Log LOG = LogFactory.getLog(this.getClass());
protected ColumnOperation(final HMaster master, final byte [] tableName)
throws IOException {
super(master, tableName);
}
@Override
protected void processScanItem(String serverName, final HRegionInfo info)
throws IOException {
if (isEnabled(info)) {
throw new TableNotDisabledException(tableName);
}
}
protected void updateRegionInfo(HRegionInterface server, byte [] regionName,
HRegionInfo i) throws IOException {
Put put = new Put(i.getRegionName());
put.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER, Writables.getBytes(i));
server.put(regionName, put);
if (LOG.isDebugEnabled()) {
LOG.debug("updated columns in row: " + i.getRegionNameAsString());
}
}
}

133
src/main/java/org/apache/hadoop/hbase/master/DeadServer.java Normal file
View File

@ -0,0 +1,133 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.util.Collection;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import org.apache.commons.lang.NotImplementedException;
import org.apache.hadoop.hbase.HServerInfo;
/**
* Class to hold dead servers list and utility querying dead server list.
*/
public class DeadServer implements Set<String> {
/**
* Set of known dead servers. On znode expiration, servers are added here.
* This is needed in case of a network partitioning where the server's lease
* expires, but the server is still running. After the network is healed,
* and it's server logs are recovered, it will be told to call server startup
* because by then, its regions have probably been reassigned.
*/
private final Set<String> deadServers = new HashSet<String>();
/**
* @param serverName
* @return true if server is dead
*/
public boolean isDeadServer(final String serverName) {
return isDeadServer(serverName, false);
}
/**
* @param serverName Servername as either <code>host:port</code> or
* <code>host,port,startcode</code>.
* @param hostAndPortOnly True if <code>serverName</code> is host and
* port only (<code>host:port</code>) and if so, then we do a prefix compare
* (ignoring start codes) looking for dead server.
* @return true if server is dead
*/
boolean isDeadServer(final String serverName, final boolean hostAndPortOnly) {
return HServerInfo.isServer(this, serverName, hostAndPortOnly);
}
public synchronized Set<String> clone() {
Set<String> clone = new HashSet<String>(this.deadServers.size());
clone.addAll(this.deadServers);
return clone;
}
public synchronized int size() {
return deadServers.size();
}
public synchronized boolean isEmpty() {
return deadServers.isEmpty();
}
public synchronized boolean contains(Object o) {
return deadServers.contains(o);
}
public Iterator<String> iterator() {
return this.deadServers.iterator();
}
public synchronized Object[] toArray() {
return deadServers.toArray();
}
public synchronized <T> T[] toArray(T[] a) {
return deadServers.toArray(a);
}
public synchronized boolean add(String e) {
return deadServers.add(e);
}
public synchronized boolean remove(Object o) {
return deadServers.remove(o);
}
public synchronized boolean containsAll(Collection<?> c) {
return deadServers.containsAll(c);
}
public synchronized boolean addAll(Collection<? extends String> c) {
return deadServers.addAll(c);
}
public synchronized boolean retainAll(Collection<?> c) {
return deadServers.retainAll(c);
}
public synchronized boolean removeAll(Collection<?> c) {
return deadServers.removeAll(c);
}
public synchronized void clear() {
throw new NotImplementedException();
}
public synchronized boolean equals(Object o) {
return deadServers.equals(o);
}
public synchronized int hashCode() {
return deadServers.hashCode();
}
public synchronized String toString() {
return this.deadServers.toString();
}
}

54
src/main/java/org/apache/hadoop/hbase/master/DeleteColumn.java
View File

@ -1,54 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.regionserver.Store;
import java.io.IOException;
/** Instantiated to remove a column family from a table */
class DeleteColumn extends ColumnOperation {
private final byte [] columnName;
DeleteColumn(final HMaster master, final byte [] tableName,
final byte [] columnName)
throws IOException {
super(master, tableName);
this.columnName = columnName;
}
@Override
protected void postProcessMeta(MetaRegion m, HRegionInterface server)
throws IOException {
for (HRegionInfo i: unservedRegions) {
i.getTableDesc().removeFamily(columnName);
updateRegionInfo(server, m.getRegionName(), i);
// Delete the directories used by the column
Path tabledir =
new Path(this.master.getRootDir(), i.getTableDesc().getNameAsString());
this.master.getFileSystem().
delete(Store.getStoreHomedir(tabledir, i.getEncodedName(),
this.columnName), true);
}
}
}

1372
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
View File

File diff suppressed because it is too large Load Diff

593
src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java Normal file
View File

@ -0,0 +1,593 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.NavigableSet;
import java.util.Random;
import java.util.TreeMap;
import java.util.TreeSet;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.fs.BlockLocation;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.io.Writable;
/**
* Makes decisions about the placement and movement of Regions across
* RegionServers.
*
* <p>Cluster-wide load balancing will occur only when there are no regions in
* transition and according to a fixed period of a time using {@link #balanceCluster(Map)}.
*
* <p>Inline region placement with {@link #immediateAssignment} can be used when
* the Master needs to handle closed regions that it currently does not have
* a destination set for. This can happen during master failover.
*
* <p>On cluster startup, {@link #bulkAssignment} can be used to determine
* locations for all Regions in a cluster.
*
* <p>This classes produces plans for the {@link AssignmentManager} to execute.
*/
public class LoadBalancer {
private static final Log LOG = LogFactory.getLog(LoadBalancer.class);
private static final Random rand = new Random();
/**
* Generate a global load balancing plan according to the specified map of
* server information to the most loaded regions of each server.
*
* The load balancing invariant is that all servers are within 1 region of the
* average number of regions per server. If the average is an integer number,
* all servers will be balanced to the average. Otherwise, all servers will
* have either floor(average) or ceiling(average) regions.
*
* The algorithm is currently implemented as such:
*
* <ol>
* <li>Determine the two valid numbers of regions each server should have,
* <b>MIN</b>=floor(average) and <b>MAX</b>=ceiling(average).
*
* <li>Iterate down the most loaded servers, shedding regions from each so
* each server hosts exactly <b>MAX</b> regions. Stop once you reach a
* server that already has &lt;= <b>MAX</b> regions.
*
* <li>Iterate down the least loaded servers, assigning regions so each server
* has exactly </b>MIN</b> regions. Stop once you reach a server that
* already has &gt;= <b>MIN</b> regions.
*
* Regions being assigned to underloaded servers are those that were shed
* in the previous step. It is possible that there were not enough
* regions shed to fill each underloaded server to <b>MIN</b>. If so we
* end up with a number of regions required to do so, <b>neededRegions</b>.
*
* It is also possible that we were able fill each underloaded but ended
* up with regions that were unassigned from overloaded servers but that
* still do not have assignment.
*
* If neither of these conditions hold (no regions needed to fill the
* underloaded servers, no regions leftover from overloaded servers),
* we are done and return. Otherwise we handle these cases below.
*
* <li>If <b>neededRegions</b> is non-zero (still have underloaded servers),
* we iterate the most loaded servers again, shedding a single server from
* each (this brings them from having <b>MAX</b> regions to having
* <b>MIN</b> regions).
*
* <li>We now definitely have more regions that need assignment, either from
* the previous step or from the original shedding from overloaded servers.
*
* Iterate the least loaded servers filling each to <b>MIN</b>.
*
* <li>If we still have more regions that need assignment, again iterate the
* least loaded servers, this time giving each one (filling them to
* </b>MAX</b>) until we run out.
*
* <li>All servers will now either host <b>MIN</b> or <b>MAX</b> regions.
*
* In addition, any server hosting &gt;= <b>MAX</b> regions is guaranteed
* to end up with <b>MAX</b> regions at the end of the balancing. This
* ensures the minimal number of regions possible are moved.
* </ol>
*
* TODO: We can at-most reassign the number of regions away from a particular
* server to be how many they report as most loaded.
* Should we just keep all assignment in memory? Any objections?
* Does this mean we need HeapSize on HMaster? Or just careful monitor?
* (current thinking is we will hold all assignments in memory)
*
* @param serverInfo map of regionservers and their load/region information to
* a list of their most loaded regions
* @return a list of regions to be moved, including source and destination,
* or null if cluster is already balanced
*/
public List<RegionPlan> balanceCluster(
Map<HServerInfo,List<HRegionInfo>> clusterState) {
LOG.debug("Running load balancer");
long startTime = System.currentTimeMillis();
// Make a map sorted by load and count regions
TreeMap<HServerInfo,List<HRegionInfo>> serversByLoad =
new TreeMap<HServerInfo,List<HRegionInfo>>(
new HServerInfo.LoadComparator());
int numServers = clusterState.size();
int numRegions = 0;
// Iterate so we can count regions as we build the map
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
clusterState.entrySet()) {
server.getKey().getLoad().setNumberOfRegions(server.getValue().size());
numRegions += server.getKey().getLoad().getNumberOfRegions();
serversByLoad.put(server.getKey(), server.getValue());
}
// Check if we even need to do any load balancing
float average = (float)numRegions / numServers; // for logging
int min = numRegions / numServers;
int max = numRegions % numServers == 0 ? min : min + 1;
if(serversByLoad.lastKey().getLoad().getNumberOfRegions() <= max &&
serversByLoad.firstKey().getLoad().getNumberOfRegions() >= min) {
// Skipped because no server outside (min,max) range
if(LOG.isDebugEnabled()) {
LOG.debug("Skipping load balancing. servers=" + numServers + " " +
"regions=" + numRegions + " average=" + average + " " +
"mostloaded=" + serversByLoad.lastKey().getLoad().getNumberOfRegions() +
" leastloaded=" + serversByLoad.lastKey().getLoad().getNumberOfRegions());
}
return null;
}
// Balance the cluster
// TODO: Look at data block locality or a more complex load to do this
List<RegionPlan> regionsToMove = new ArrayList<RegionPlan>();
int regionidx = 0; // track the index in above list for setting destination
// Walk down most loaded, pruning each to the max
int serversOverloaded = 0;
Map<HServerInfo,BalanceInfo> serverBalanceInfo =
new TreeMap<HServerInfo,BalanceInfo>();
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.descendingMap().entrySet()) {
HServerInfo serverInfo = server.getKey();
int regionCount = serverInfo.getLoad().getNumberOfRegions();
if(regionCount <= max) {
serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
break;
}
serversOverloaded++;
List<HRegionInfo> regions = server.getValue();
int numToOffload = Math.min(regionCount - max, regions.size());
int numTaken = 0;
for (HRegionInfo hri: regions) {
// Don't rebalance meta regions.
if (hri.isMetaRegion()) continue;
regionsToMove.add(new RegionPlan(hri, serverInfo, null));
numTaken++;
if (numTaken >= numToOffload) break;
}
serverBalanceInfo.put(serverInfo,
new BalanceInfo(numToOffload, (-1)*numTaken));
}
// Walk down least loaded, filling each to the min
int serversUnderloaded = 0; // number of servers that get new regions
int neededRegions = 0; // number of regions needed to bring all up to min
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.entrySet()) {
int regionCount = server.getKey().getLoad().getNumberOfRegions();
if(regionCount >= min) {
break;
}
serversUnderloaded++;
int numToTake = min - regionCount;
int numTaken = 0;
while(numTaken < numToTake && regionidx < regionsToMove.size()) {
regionsToMove.get(regionidx).setDestination(server.getKey());
numTaken++;
regionidx++;
}
serverBalanceInfo.put(server.getKey(), new BalanceInfo(0, numTaken));
// If we still want to take some, increment needed
if(numTaken < numToTake) {
neededRegions += (numToTake - numTaken);
}
}
// If none needed to fill all to min and none left to drain all to max,
// we are done
if(neededRegions == 0 && regionidx == regionsToMove.size()) {
long endTime = System.currentTimeMillis();
LOG.info("Calculated a load balance in " + (endTime-startTime) + "ms. " +
"Moving " + regionsToMove.size() + " regions off of " +
serversOverloaded + " overloaded servers onto " +
serversUnderloaded + " less loaded servers");
return regionsToMove;
}
// Need to do a second pass.
// Either more regions to assign out or servers that are still underloaded
// If we need more to fill min, grab one from each most loaded until enough
if(neededRegions != 0) {
// Walk down most loaded, grabbing one from each until we get enough
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.descendingMap().entrySet()) {
BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
int idx =
balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
HRegionInfo region = server.getValue().get(idx);
if (region.isMetaRegion()) continue; // Don't move meta regions.
regionsToMove.add(new RegionPlan(region, server.getKey(), null));
if(--neededRegions == 0) {
// No more regions needed, done shedding
break;
}
}
}
// Now we have a set of regions that must be all assigned out
// Assign each underloaded up to the min, then if leftovers, assign to max
// Walk down least loaded, assigning to each to fill up to min
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.entrySet()) {
int regionCount = server.getKey().getLoad().getNumberOfRegions();
BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
if(balanceInfo != null) {
regionCount += balanceInfo.getNumRegionsAdded();
}
if(regionCount >= min) {
break;
}
int numToTake = min - regionCount;
int numTaken = 0;
while(numTaken < numToTake && regionidx < regionsToMove.size()) {
regionsToMove.get(regionidx).setDestination(server.getKey());
numTaken++;
regionidx++;
}
}
// If we still have regions to dish out, assign underloaded to max
if(regionidx != regionsToMove.size()) {
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.entrySet()) {
int regionCount = server.getKey().getLoad().getNumberOfRegions();
if(regionCount >= max) {
break;
}
regionsToMove.get(regionidx).setDestination(server.getKey());
regionidx++;
if(regionidx == regionsToMove.size()) {
break;
}
}
}
long endTime = System.currentTimeMillis();
assert(regionidx == regionsToMove.size());
assert(neededRegions == 0);
// All done!
LOG.info("Calculated a load balance in " + (endTime-startTime) + "ms. " +
"Moving " + regionsToMove.size() + " regions off of " +
serversOverloaded + " overloaded servers onto " +
serversUnderloaded + " less loaded servers");
return regionsToMove;
}
/**
* Stores additional per-server information about the regions added/removed
* during the run of the balancing algorithm.
*
* For servers that receive additional regions, we are not updating the number
* of regions in HServerInfo once we decide to reassign regions to a server,
* but we need this information later in the algorithm. This is stored in
* <b>numRegionsAdded</b>.
*
* For servers that shed regions, we need to track which regions we have
* already shed. <b>nextRegionForUnload</b> contains the index in the list
* of regions on the server that is the next to be shed.
*/
private static class BalanceInfo {
private final int nextRegionForUnload;
private final int numRegionsAdded;
public BalanceInfo(int nextRegionForUnload, int numRegionsAdded) {
this.nextRegionForUnload = nextRegionForUnload;
this.numRegionsAdded = numRegionsAdded;
}
public int getNextRegionForUnload() {
return nextRegionForUnload;
}
public int getNumRegionsAdded() {
return numRegionsAdded;
}
}
/**
* Generates a bulk assignment plan to be used on cluster startup.
*
* Takes a list of all the regions and all the servers in the cluster and
* returns a map of each server to the regions that it should be assigned.
*
* Currently implemented as a round-robin assignment. Same invariant as
* load balancing, all servers holding floor(avg) or ceiling(avg).
*
* TODO: Use block locations from HDFS to place regions with their blocks
*
* @param regions all regions
* @param servers all servers
* @return map of server to the regions it should take, or null if no
* assignment is possible (ie. no regions or no servers)
*/
public static Map<HServerInfo,List<HRegionInfo>> bulkAssignment(
List<HRegionInfo> regions, List<HServerInfo> servers) {
if(regions.size() == 0 || servers.size() == 0) {
return null;
}
Map<HServerInfo,List<HRegionInfo>> assignments =
new TreeMap<HServerInfo,List<HRegionInfo>>();
int numRegions = regions.size();
int numServers = servers.size();
int max = (int)Math.ceil((float)numRegions/numServers);
int serverIdx = 0;
for(HServerInfo server : servers) {
List<HRegionInfo> serverRegions = new ArrayList<HRegionInfo>(max);
for(int i=serverIdx;i<regions.size();i+=numServers) {
serverRegions.add(regions.get(i));
}
assignments.put(server, serverRegions);
serverIdx++;
}
return assignments;
}
/**
* Find the block locations for all of the files for the specified region.
*
* Returns an ordered list of hosts that are hosting the blocks for this
* region. The weight of each host is the sum of the block lengths of all
* files on that host, so the first host in the list is the server which
* holds the most bytes of the given region's HFiles.
*
* TODO: Make this work. Need to figure out how to match hadoop's hostnames
* given for block locations with our HServerAddress.
* TODO: Use the right directory for the region
* TODO: Use getFileBlockLocations on the files not the directory
*
* @param fs the filesystem
* @param region region
* @return ordered list of hosts holding blocks of the specified region
* @throws IOException if any filesystem errors
*/
private List<String> getTopBlockLocations(FileSystem fs, HRegionInfo region)
throws IOException {
String encodedName = region.getEncodedName();
Path path = new Path("/hbase/table/" + encodedName);
FileStatus status = fs.getFileStatus(path);
BlockLocation [] blockLocations =
fs.getFileBlockLocations(status, 0, status.getLen());
Map<HostAndWeight,HostAndWeight> hostWeights =
new TreeMap<HostAndWeight,HostAndWeight>(new HostAndWeight.HostComparator());
for(BlockLocation bl : blockLocations) {
String [] hosts = bl.getHosts();
long len = bl.getLength();
for(String host : hosts) {
HostAndWeight haw = hostWeights.get(host);
if(haw == null) {
haw = new HostAndWeight(host, len);
hostWeights.put(haw, haw);
} else {
haw.addWeight(len);
}
}
}
NavigableSet<HostAndWeight> orderedHosts = new TreeSet<HostAndWeight>(
new HostAndWeight.WeightComparator());
orderedHosts.addAll(hostWeights.values());
List<String> topHosts = new ArrayList<String>(orderedHosts.size());
for(HostAndWeight haw : orderedHosts.descendingSet()) {
topHosts.add(haw.getHost());
}
return topHosts;
}
/**
* Stores the hostname and weight for that hostname.
*
* This is used when determining the physical locations of the blocks making
* up a region.
*
* To make a prioritized list of the hosts holding the most data of a region,
* this class is used to count the total weight for each host. The weight is
* currently just the size of the file.
*/
private static class HostAndWeight {
private final String host;
private long weight;
public HostAndWeight(String host, long weight) {
this.host = host;
this.weight = weight;
}
public void addWeight(long weight) {
this.weight += weight;
}
public String getHost() {
return host;
}
public long getWeight() {
return weight;
}
private static class HostComparator implements Comparator<HostAndWeight> {
@Override
public int compare(HostAndWeight l, HostAndWeight r) {
return l.getHost().compareTo(r.getHost());
}
}
private static class WeightComparator implements Comparator<HostAndWeight> {
@Override
public int compare(HostAndWeight l, HostAndWeight r) {
if(l.getWeight() == r.getWeight()) {
return l.getHost().compareTo(r.getHost());
}
return l.getWeight() < r.getWeight() ? -1 : 1;
}
}
}
/**
* Generates an immediate assignment plan to be used by a new master for
* regions in transition that do not have an already known destination.
*
* Takes a list of regions that need immediate assignment and a list of
* all available servers. Returns a map of regions to the server they
* should be assigned to.
*
* This method will return quickly and does not do any intelligent
* balancing. The goal is to make a fast decision not the best decision
* possible.
*
* Currently this is random.
*
* @param regions
* @param servers
* @return map of regions to the server it should be assigned to
*/
public static Map<HRegionInfo,HServerInfo> immediateAssignment(
List<HRegionInfo> regions, List<HServerInfo> servers) {
Map<HRegionInfo,HServerInfo> assignments =
new TreeMap<HRegionInfo,HServerInfo>();
for(HRegionInfo region : regions) {
assignments.put(region, servers.get(rand.nextInt(servers.size())));
}
return assignments;
}
public static HServerInfo randomAssignment(List<HServerInfo> servers) {
if (servers == null || servers.isEmpty()) {
LOG.warn("Wanted to do random assignment but no servers to assign to");
return null;
}
return servers.get(rand.nextInt(servers.size()));
}
/**
* Stores the plan for the move of an individual region.
*
* Contains info for the region being moved, info for the server the region
* should be moved from, and info for the server the region should be moved
* to.
*
* The comparable implementation of this class compares only the region
* information and not the source/dest server info.
*/
public static class RegionPlan implements Comparable<RegionPlan> {
private final HRegionInfo hri;
private final HServerInfo source;
private HServerInfo dest;
/**
* Instantiate a plan for a region move, moving the specified region from
* the specified source server to the specified destination server.
*
* Destination server can be instantiated as null and later set
* with {@link #setDestination(HServerInfo)}.
*
* @param hri region to be moved
* @param source regionserver region should be moved from
* @param dest regionserver region should be moved to
*/
public RegionPlan(final HRegionInfo hri, HServerInfo source, HServerInfo dest) {
this.hri = hri;
this.source = source;
this.dest = dest;
}
/**
* Set the destination server for the plan for this region.
*/
public void setDestination(HServerInfo dest) {
this.dest = dest;
}
/**
* Get the source server for the plan for this region.
* @return server info for source
*/
public HServerInfo getSource() {
return source;
}
/**
* Get the destination server for the plan for this region.
* @return server info for destination
*/
public HServerInfo getDestination() {
return dest;
}
/**
* Get the encoded region name for the region this plan is for.
* @return Encoded region name
*/
public String getRegionName() {
return this.hri.getEncodedName();
}
public HRegionInfo getRegionInfo() {
return this.hri;
}
/**
* Compare the region info.
* @param o region plan you are comparing against
*/
@Override
public int compareTo(RegionPlan o) {
return getRegionName().compareTo(o.getRegionName());
}
}
}

24
src/main/java/org/apache/hadoop/hbase/master/LogsCleaner.java → src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
View File

@ -19,6 +19,10 @@
*/
package org.apache.hadoop.hbase.master;
import java.io.IOException;
import java.util.LinkedList;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
@ -26,23 +30,17 @@ import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.Chore;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.Stoppable;
import org.apache.hadoop.hbase.regionserver.wal.HLog;
import java.io.IOException;
import java.util.LinkedList;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
/**
* This Chore, everytime it runs, will clear the logs in the old logs folder
* This Chore, everytime it runs, will clear the wal logs in the old logs folder
* that are deletable for each log cleaner in the chain, in order to limit the
* number of deletes it sends, will only delete maximum 20 in a single run.
*/
public class LogsCleaner extends Chore {
static final Log LOG = LogFactory.getLog(LogsCleaner.class.getName());
public class LogCleaner extends Chore {
static final Log LOG = LogFactory.getLog(LogCleaner.class.getName());
// Max number we can delete on every chore, this is to make sure we don't
// issue thousands of delete commands around the same time
@ -55,12 +53,12 @@ public class LogsCleaner extends Chore {
/**
*
* @param p the period of time to sleep between each run
* @param s the stopper boolean
* @param s the stopper
* @param conf configuration to use
* @param fs handle to the FS
* @param oldLogDir the path to the archived logs
*/
public LogsCleaner(final int p, final AtomicBoolean s,
public LogCleaner(final int p, final Stoppable s,
Configuration conf, FileSystem fs,
Path oldLogDir) {
super("LogsCleaner", p, s);
@ -154,4 +152,4 @@ public class LogsCleaner extends Chore {
LOG.warn("Error while cleaning the logs", e);
}
}
}
}

3
src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
View File

@ -45,5 +45,4 @@ public interface LogCleanerDelegate extends Configurable {
* @return true if the log is deletable, false if not
*/
public boolean isLogDeletable(Path filePath);
}
}

279
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java Normal file
View File

@ -0,0 +1,279 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.io.IOException;
import java.util.Map;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.Store;
import org.apache.hadoop.hbase.regionserver.wal.HLog;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.FSUtils;
/**
* This class abstracts a bunch of operations the HMaster needs to interact with
* the underlying file system, including splitting log files, checking file
* system status, etc.
*/
public class MasterFileSystem {
private static final Log LOG = LogFactory.getLog(MasterFileSystem.class.getName());
// HBase configuration
Configuration conf;
// master status
Server master;
// Keep around for convenience.
private final FileSystem fs;
// Is the fileystem ok?
private volatile boolean fsOk = true;
// The Path to the old logs dir
private final Path oldLogDir;
// root hbase directory on the FS
private final Path rootdir;
// create the split log lock
final Lock splitLogLock = new ReentrantLock();
public MasterFileSystem(Server master) throws IOException {
this.conf = master.getConfiguration();
this.master = master;
// Set filesystem to be that of this.rootdir else we get complaints about
// mismatched filesystems if hbase.rootdir is hdfs and fs.defaultFS is
// default localfs. Presumption is that rootdir is fully-qualified before
// we get to here with appropriate fs scheme.
this.rootdir = FSUtils.getRootDir(conf);
// Cover both bases, the old way of setting default fs and the new.
// We're supposed to run on 0.20 and 0.21 anyways.
conf.set("fs.default.name", this.rootdir.toString());
conf.set("fs.defaultFS", this.rootdir.toString());
// setup the filesystem variable
this.fs = FileSystem.get(conf);
// set up the archived logs path
this.oldLogDir = new Path(this.rootdir, HConstants.HREGION_OLDLOGDIR_NAME);
}
/**
* <ol>
* <li>Check if the root region exists and is readable, if not create it</li>
* <li>Create a log archive directory for RS to put archived logs</li>
* </ol>
*/
public void initialize() throws IOException {
// check if the root directory exists
checkRootDir(this.rootdir, conf, this.fs);
// Make sure the region servers can archive their old logs
if(!this.fs.exists(this.oldLogDir)) {
this.fs.mkdirs(this.oldLogDir);
}
}
public FileSystem getFileSystem() {
return this.fs;
}
/**
* Get the directory where old logs go
* @return the dir
*/
public Path getOldLogDir() {
return this.oldLogDir;
}
/**
* Checks to see if the file system is still accessible.
* If not, sets closed
* @return false if file system is not available
*/
public boolean checkFileSystem() {
if (this.fsOk) {
try {
FSUtils.checkFileSystemAvailable(this.fs);
} catch (IOException e) {
master.abort("Shutting down HBase cluster: file system not available", e);
this.fsOk = false;
}
}
return this.fsOk;
}
/**
* @return HBase root dir.
* @throws IOException
*/
public Path getRootDir() {
return this.rootdir;
}
/**
* Inspect the log directory to recover any log file without
* an active region server.
* @param onlineServers Map of online servers keyed by
* {@link HServerInfo#getServerName()}
*/
void splitLogAfterStartup(final Map<String, HServerInfo> onlineServers) {
Path logsDirPath = new Path(this.rootdir, HConstants.HREGION_LOGDIR_NAME);
try {
if (!this.fs.exists(logsDirPath)) {
return;
}
} catch (IOException e) {
throw new RuntimeException("Failed exists test on " + logsDirPath, e);
}
FileStatus[] logFolders;
try {
logFolders = this.fs.listStatus(logsDirPath);
} catch (IOException e) {
throw new RuntimeException("Failed listing " + logsDirPath.toString(), e);
}
if (logFolders == null || logFolders.length == 0) {
LOG.debug("No log files to split, proceeding...");
return;
}
for (FileStatus status : logFolders) {
String serverName = status.getPath().getName();
LOG.info("Found log folder : " + serverName);
if(onlineServers.get(serverName) == null) {
LOG.info("Log folder doesn't belong " +
"to a known region server, splitting");
splitLog(serverName);
} else {
LOG.info("Log folder belongs to an existing region server");
}
}
}
public void splitLog(final String serverName) {
this.splitLogLock.lock();
Path logDir = new Path(this.rootdir, HLog.getHLogDirectoryName(serverName));
try {
HLog.splitLog(this.rootdir, logDir, oldLogDir, this.fs, conf);
} catch (IOException e) {
LOG.error("Failed splitting " + logDir.toString(), e);
} finally {
this.splitLogLock.unlock();
}
}
/**
* Get the rootdir. Make sure its wholesome and exists before returning.
* @param rd
* @param conf
* @param fs
* @return hbase.rootdir (after checks for existence and bootstrapping if
* needed populating the directory with necessary bootup files).
* @throws IOException
*/
private static Path checkRootDir(final Path rd, final Configuration c,
final FileSystem fs)
throws IOException {
// If FS is in safe mode wait till out of it.
FSUtils.waitOnSafeMode(c, c.getInt(HConstants.THREAD_WAKE_FREQUENCY,
10 * 1000));
// Filesystem is good. Go ahead and check for hbase.rootdir.
if (!fs.exists(rd)) {
fs.mkdirs(rd);
FSUtils.setVersion(fs, rd);
} else {
FSUtils.checkVersion(fs, rd, true);
}
// Make sure the root region directory exists!
if (!FSUtils.rootRegionExists(fs, rd)) {
bootstrap(rd, c);
}
return rd;
}
private static void bootstrap(final Path rd, final Configuration c)
throws IOException {
LOG.info("BOOTSTRAP: creating ROOT and first META regions");
try {
// Bootstrapping, make sure blockcache is off. Else, one will be
// created here in bootstap and it'll need to be cleaned up. Better to
// not make it in first place. Turn off block caching for bootstrap.
// Enable after.
HRegionInfo rootHRI = new HRegionInfo(HRegionInfo.ROOT_REGIONINFO);
setInfoFamilyCaching(rootHRI, false);
HRegionInfo metaHRI = new HRegionInfo(HRegionInfo.FIRST_META_REGIONINFO);
setInfoFamilyCaching(metaHRI, false);
HRegion root = HRegion.createHRegion(rootHRI, rd, c);
HRegion meta = HRegion.createHRegion(metaHRI, rd, c);
setInfoFamilyCaching(rootHRI, true);
setInfoFamilyCaching(metaHRI, true);
// Add first region from the META table to the ROOT region.
HRegion.addRegionToMETA(root, meta);
root.close();
root.getLog().closeAndDelete();
meta.close();
meta.getLog().closeAndDelete();
} catch (IOException e) {
e = RemoteExceptionHandler.checkIOException(e);
LOG.error("bootstrap", e);
throw e;
}
}
/**
* @param hri Set all family block caching to <code>b</code>
* @param b
*/
private static void setInfoFamilyCaching(final HRegionInfo hri, final boolean b) {
for (HColumnDescriptor hcd: hri.getTableDesc().families.values()) {
if (Bytes.equals(hcd.getName(), HConstants.CATALOG_FAMILY)) {
hcd.setBlockCacheEnabled(b);
hcd.setInMemory(b);
}
}
}
public void deleteRegion(HRegionInfo region) throws IOException {
fs.delete(HRegion.getRegionDir(rootdir, region), true);
}
public void deleteTable(byte[] tableName) throws IOException {
fs.delete(new Path(rootdir, Bytes.toString(tableName)), true);
}
public void updateRegionInfo(HRegionInfo region) {
// TODO implement this. i think this is currently broken in trunk i don't
// see this getting updated.
// @see HRegion.checkRegioninfoOnFilesystem()
}
public void deleteFamily(HRegionInfo region, byte[] familyName)
throws IOException {
fs.delete(Store.getStoreHomedir(
new Path(rootdir, region.getTableDesc().getNameAsString()),
region.getEncodedName(), familyName), true);
}
}

59
src/main/java/org/apache/hadoop/hbase/master/MasterServices.java Normal file
View File

@ -0,0 +1,59 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.io.IOException;
import org.apache.hadoop.hbase.TableNotDisabledException;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.executor.ExecutorService;
/**
* Services Master supplies
*/
public interface MasterServices {
/**
* @return Master's instance of the {@link AssignmentManager}
*/
public AssignmentManager getAssignmentManager();
/**
* @return Master's filesystem {@link MasterFileSystem} utility class.
*/
public MasterFileSystem getMasterFileSystem();
/**
* @return Master's {@link ServerManager} instance.
*/
public ServerManager getServerManager();
/**
* @return Master's instance of {@link ExecutorService}
*/
public ExecutorService getExecutorService();
/**
* Check table is modifiable; i.e. exists and is offline.
* @param tableName Name of table to check.
* @throws TableNotDisabledException
* @throws TableNotFoundException
*/
public void checkTableModifiable(final byte [] tableName) throws IOException;
}

96
src/main/java/org/apache/hadoop/hbase/master/MetaRegion.java
View File

@ -1,96 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.util.Bytes;
/** Describes a meta region and its server */
public class MetaRegion implements Comparable<MetaRegion> {
private final HServerAddress server;
private HRegionInfo regionInfo;
MetaRegion(final HServerAddress server, HRegionInfo regionInfo) {
if (server == null) {
throw new IllegalArgumentException("server cannot be null");
}
this.server = server;
if (regionInfo == null) {
throw new IllegalArgumentException("regionInfo cannot be null");
}
this.regionInfo = regionInfo;
}
@Override
public String toString() {
return "{server: " + this.server.toString() + ", regionname: " +
regionInfo.getRegionNameAsString() + ", startKey: <" +
Bytes.toString(regionInfo.getStartKey()) + ">}";
}
/** @return the regionName */
public byte [] getRegionName() {
return regionInfo.getRegionName();
}
/** @return the server */
public HServerAddress getServer() {
return server;
}
/** @return the startKey */
public byte [] getStartKey() {
return regionInfo.getStartKey();
}
/** @return the endKey */
public byte [] getEndKey() {
return regionInfo.getEndKey();
}
public HRegionInfo getRegionInfo() {
return regionInfo;
}
@Override
public boolean equals(Object o) {
return o instanceof MetaRegion && this.compareTo((MetaRegion)o) == 0;
}
@Override
public int hashCode() {
return regionInfo.hashCode();
}
// Comparable
public int compareTo(MetaRegion other) {
int cmp = regionInfo.compareTo(other.regionInfo);
if(cmp == 0) {
// Might be on different host?
cmp = this.server.compareTo(other.server);
}
return cmp;
}
}

181
src/main/java/org/apache/hadoop/hbase/master/MetaScanner.java
View File

@ -1,181 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
/**
* MetaScanner <code>META</code> table.
*
* When a <code>META</code> server comes on line, a MetaRegion object is
* queued up by regionServerReport() and this thread wakes up.
*
* It's important to do this work in a separate thread, or else the blocking
* action would prevent other work from getting done.
*/
class MetaScanner extends BaseScanner {
/** Initial work for the meta scanner is queued up here */
private volatile BlockingQueue<MetaRegion> metaRegionsToScan =
new LinkedBlockingQueue<MetaRegion>();
private final List<MetaRegion> metaRegionsToRescan =
new ArrayList<MetaRegion>();
/**
* Constructor
*
* @param master
*/
public MetaScanner(HMaster master) {
super(master, false, master.getShutdownRequested());
}
// Don't retry if we get an error while scanning. Errors are most often
// caused by the server going away. Wait until next rescan interval when
// things should be back to normal.
private boolean scanOneMetaRegion(MetaRegion region) {
while (!this.master.isClosed() &&
!this.master.getRegionManager().isInitialRootScanComplete() &&
this.master.getRegionManager().getRootRegionLocation() == null) {
sleep();
}
if (this.master.isClosed()) {
return false;
}
try {
// Don't interrupt us while we're working
synchronized (scannerLock) {
scanRegion(region);
this.master.getRegionManager().putMetaRegionOnline(region);
}
} catch (IOException e) {
e = RemoteExceptionHandler.checkIOException(e);
LOG.warn("Scan one META region: " + region.toString(), e);
// The region may have moved (TestRegionServerAbort, etc.). If
// so, either it won't be in the onlineMetaRegions list or its host
// address has changed and the containsValue will fail. If not
// found, best thing to do here is probably return.
if (!this.master.getRegionManager().isMetaRegionOnline(region.getStartKey())) {
LOG.debug("Scanned region is no longer in map of online " +
"regions or its value has changed");
return false;
}
// Make sure the file system is still available
this.master.checkFileSystem();
} catch (Exception e) {
// If for some reason we get some other kind of exception,
// at least log it rather than go out silently.
LOG.error("Unexpected exception", e);
}
return true;
}
@Override
protected boolean initialScan() {
MetaRegion region = null;
while (!this.master.isClosed() &&
(region == null && metaRegionsToScan.size() > 0) &&
!metaRegionsScanned()) {
try {
region = metaRegionsToScan.poll(this.master.getThreadWakeFrequency(),
TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
// continue
}
if (region == null && metaRegionsToRescan.size() != 0) {
region = metaRegionsToRescan.remove(0);
}
if (region != null) {
if (!scanOneMetaRegion(region)) {
metaRegionsToRescan.add(region);
}
}
}
initialScanComplete = true;
return true;
}
@Override
protected void maintenanceScan() {
List<MetaRegion> regions =
this.master.getRegionManager().getListOfOnlineMetaRegions();
int regionCount = 0;
for (MetaRegion r: regions) {
scanOneMetaRegion(r);
regionCount++;
}
LOG.info("All " + regionCount + " .META. region(s) scanned");
metaRegionsScanned();
}
/*
* Called by the meta scanner when it has completed scanning all meta
* regions. This wakes up any threads that were waiting for this to happen.
* @param totalRows Total rows scanned.
* @param regionCount Count of regions in .META. table.
* @return False if number of meta regions matches count of online regions.
*/
private synchronized boolean metaRegionsScanned() {
if (!this.master.getRegionManager().isInitialRootScanComplete() ||
this.master.getRegionManager().numMetaRegions() !=
this.master.getRegionManager().numOnlineMetaRegions()) {
return false;
}
notifyAll();
return true;
}
/**
* Other threads call this method to wait until all the meta regions have
* been scanned.
*/
synchronized boolean waitForMetaRegionsOrClose() {
while (!this.master.isClosed()) {
synchronized (master.getRegionManager()) {
if (this.master.getRegionManager().isInitialRootScanComplete() &&
this.master.getRegionManager().numMetaRegions() ==
this.master.getRegionManager().numOnlineMetaRegions()) {
break;
}
}
try {
wait(this.master.getThreadWakeFrequency());
} catch (InterruptedException e) {
// continue
}
}
return this.master.isClosed();
}
/**
* Add another meta region to scan to the queue.
*/
void addMetaRegionToScan(MetaRegion m) {
metaRegionsToScan.add(m);
}
}

56
src/main/java/org/apache/hadoop/hbase/master/ModifyColumn.java
View File

@ -1,56 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
/** Instantiated to modify an existing column family on a table */
class ModifyColumn extends ColumnOperation {
private final HColumnDescriptor descriptor;
private final byte [] columnName;
ModifyColumn(final HMaster master, final byte [] tableName,
final byte [] columnName, HColumnDescriptor descriptor)
throws IOException {
super(master, tableName);
this.descriptor = descriptor;
this.columnName = columnName;
}
@Override
protected void postProcessMeta(MetaRegion m, HRegionInterface server)
throws IOException {
for (HRegionInfo i: unservedRegions) {
if (i.getTableDesc().hasFamily(columnName)) {
i.getTableDesc().addFamily(descriptor);
updateRegionInfo(server, m.getRegionName(), i);
} else { // otherwise, we have an error.
throw new InvalidColumnNameException("Column family '" +
Bytes.toString(columnName) +
"' doesn't exist, so cannot be modified.");
}
}
}
}

78
src/main/java/org/apache/hadoop/hbase/master/ModifyTableMeta.java
View File

@ -1,78 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableNotDisabledException;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Writables;
import java.io.IOException;
/** Instantiated to modify table descriptor metadata */
class ModifyTableMeta extends TableOperation {
private static Log LOG = LogFactory.getLog(ModifyTableMeta.class);
private HTableDescriptor desc;
ModifyTableMeta(final HMaster master, final byte [] tableName,
HTableDescriptor desc)
throws IOException {
super(master, tableName);
this.desc = desc;
LOG.debug("modifying " + Bytes.toString(tableName) + ": " +
desc.toString());
}
protected void updateRegionInfo(HRegionInterface server, byte [] regionName,
HRegionInfo i)
throws IOException {
Put put = new Put(i.getRegionName());
put.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER, Writables.getBytes(i));
server.put(regionName, put);
LOG.debug("updated HTableDescriptor for region " + i.getRegionNameAsString());
}
@Override
protected void processScanItem(String serverName,
final HRegionInfo info) throws IOException {
if (isEnabled(info)) {
throw new TableNotDisabledException(Bytes.toString(tableName));
}
}
@Override
protected void postProcessMeta(MetaRegion m, HRegionInterface server)
throws IOException {
for (HRegionInfo i: unservedRegions) {
i.setTableDesc(desc);
updateRegionInfo(server, m.getRegionName(), i);
}
// kick off a meta scan right away
master.getRegionManager().metaScannerThread.triggerNow();
}
}

109
src/main/java/org/apache/hadoop/hbase/master/ProcessRegionClose.java
View File

@ -1,109 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.regionserver.HRegion;
import java.io.IOException;
/**
* ProcessRegionClose is the way we do post-processing on a closed region. We
* only spawn one of these asynchronous tasks when the region needs to be
* either offlined or deleted. We used to create one of these tasks whenever
* a region was closed, but since closing a region that isn't being offlined
* or deleted doesn't actually require post processing, it's no longer
* necessary.
*/
public class ProcessRegionClose extends ProcessRegionStatusChange {
protected final boolean offlineRegion;
protected final boolean reassignRegion;
/**
* @param master
* @param regionInfo Region to operate on
* @param offlineRegion if true, set the region to offline in meta
* @param reassignRegion if true, region is to be reassigned
*/
public ProcessRegionClose(HMaster master, HRegionInfo regionInfo,
boolean offlineRegion, boolean reassignRegion) {
super(master, regionInfo);
this.offlineRegion = offlineRegion;
this.reassignRegion = reassignRegion;
}
@Override
public String toString() {
return "ProcessRegionClose of " + this.regionInfo.getRegionNameAsString() +
", " + this.offlineRegion + ", reassign: " + this.reassignRegion;
}
@Override
protected boolean process() throws IOException {
if (!metaRegionAvailable()) {
// We can't proceed unless the meta region we are going to update
// is online. metaRegionAvailable() has put this operation on the
// delayedToDoQueue, so return true so the operation is not put
// back on the toDoQueue
return true;
}
Boolean result = null;
if (offlineRegion || reassignRegion) {
result =
new RetryableMetaOperation<Boolean>(getMetaRegion(), this.master) {
public Boolean call() throws IOException {
// We can't proceed unless the meta region we are going to update
// is online. metaRegionAvailable() will put this operation on the
// delayedToDoQueue, so return true so the operation is not put
// back on the toDoQueue
if (metaRegionAvailable()) {
if(offlineRegion) {
// offline the region in meta and then remove it from the
// set of regions in transition
HRegion.offlineRegionInMETA(server, metaRegionName,
regionInfo);
master.getRegionManager().removeRegion(regionInfo);
LOG.info("region closed: " + regionInfo.getRegionNameAsString());
} else {
// we are reassigning the region eventually, so set it unassigned
// and remove the server info
HRegion.cleanRegionInMETA(server, metaRegionName,
regionInfo);
master.getRegionManager().setUnassigned(regionInfo, false);
LOG.info("region set as unassigned: " + regionInfo.getRegionNameAsString());
}
}
return true;
}
}.doWithRetries();
result = result == null ? true : result;
} else {
LOG.info("Region was neither offlined, or asked to be reassigned, what gives: " +
regionInfo.getRegionNameAsString());
}
return result == null ? true : result;
}
}

130
src/main/java/org/apache/hadoop/hbase/master/ProcessRegionOpen.java
View File

@ -1,130 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper;
import java.io.IOException;
/**
* ProcessRegionOpen is instantiated when a region server reports that it is
* serving a region. This applies to all meta and user regions except the
* root region which is handled specially.
*/
public class ProcessRegionOpen extends ProcessRegionStatusChange {
protected final HServerInfo serverInfo;
/**
* @param master
* @param info
* @param regionInfo
*/
public ProcessRegionOpen(HMaster master, HServerInfo info,
HRegionInfo regionInfo) {
super(master, regionInfo);
if (info == null) {
throw new NullPointerException("HServerInfo cannot be null; " +
"hbase-958 debugging");
}
this.serverInfo = info;
}
@Override
public String toString() {
return "PendingOpenOperation from " + serverInfo.getServerName();
}
@Override
protected boolean process() throws IOException {
// TODO: The below check is way too convoluted!!!
if (!metaRegionAvailable()) {
// We can't proceed unless the meta region we are going to update
// is online. metaRegionAvailable() has put this operation on the
// delayedToDoQueue, so return true so the operation is not put
// back on the toDoQueue
return true;
}
HRegionInterface server =
master.getServerConnection().getHRegionConnection(getMetaRegion().getServer());
LOG.info(regionInfo.getRegionNameAsString() + " open on " +
serverInfo.getServerName());
// Register the newly-available Region's location.
Put p = new Put(regionInfo.getRegionName());
p.add(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER,
Bytes.toBytes(serverInfo.getHostnamePort()));
p.add(HConstants.CATALOG_FAMILY, HConstants.STARTCODE_QUALIFIER,
Bytes.toBytes(serverInfo.getStartCode()));
server.put(metaRegionName, p);
LOG.info("Updated row " + regionInfo.getRegionNameAsString() +
" in region " + Bytes.toString(metaRegionName) + " with startcode=" +
serverInfo.getStartCode() + ", server=" + serverInfo.getHostnamePort());
synchronized (master.getRegionManager()) {
if (isMetaTable) {
// It's a meta region.
MetaRegion m =
new MetaRegion(new HServerAddress(serverInfo.getServerAddress()),
regionInfo);
if (!master.getRegionManager().isInitialMetaScanComplete()) {
// Put it on the queue to be scanned for the first time.
if (LOG.isDebugEnabled()) {
LOG.debug("Adding " + m.toString() + " to regions to scan");
}
master.getRegionManager().addMetaRegionToScan(m);
} else {
// Add it to the online meta regions
if (LOG.isDebugEnabled()) {
LOG.debug("Adding to onlineMetaRegions: " + m.toString());
}
master.getRegionManager().putMetaRegionOnline(m);
// Interrupting the Meta Scanner sleep so that it can
// process regions right away
master.getRegionManager().metaScannerThread.triggerNow();
}
}
// If updated successfully, remove from pending list if the state
// is consistent. For example, a disable could be called before the
// synchronization.
if(master.getRegionManager().
isOfflined(regionInfo.getRegionNameAsString())) {
LOG.warn("We opened a region while it was asked to be closed.");
} else {
master.getRegionManager().removeRegion(regionInfo);
}
ZooKeeperWrapper zkWrapper =
ZooKeeperWrapper.getInstance(master.getConfiguration(),
HMaster.class.getName());
zkWrapper.deleteUnassignedRegion(regionInfo.getEncodedName());
return true;
}
}
@Override
protected int getPriority() {
return 0; // highest priority
}
}

85
src/main/java/org/apache/hadoop/hbase/master/ProcessRegionStatusChange.java
View File

@ -1,85 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.HRegionInfo;
/**
* Abstract class that performs common operations for
* @see ProcessRegionClose and @see ProcessRegionOpen
*/
abstract class ProcessRegionStatusChange extends RegionServerOperation {
protected final boolean isMetaTable;
protected final HRegionInfo regionInfo;
@SuppressWarnings({"FieldCanBeLocal"})
private volatile MetaRegion metaRegion = null;
protected volatile byte[] metaRegionName = null;
/**
* @param master the master
* @param regionInfo region info
*/
public ProcessRegionStatusChange(HMaster master, HRegionInfo regionInfo) {
super(master);
this.regionInfo = regionInfo;
this.isMetaTable = regionInfo.isMetaTable();
}
protected boolean metaRegionAvailable() {
boolean available = true;
if (isMetaTable) {
// This operation is for the meta table
if (!rootAvailable()) {
requeue();
// But we can't proceed unless the root region is available
available = false;
}
} else {
if (!master.getRegionManager().isInitialRootScanComplete() ||
!metaTableAvailable()) {
// The root region has not been scanned or the meta table is not
// available so we can't proceed.
// Put the operation on the delayedToDoQueue
requeue();
available = false;
}
}
return available;
}
protected MetaRegion getMetaRegion() {
if (isMetaTable) {
this.metaRegionName = HRegionInfo.ROOT_REGIONINFO.getRegionName();
this.metaRegion = new MetaRegion(master.getRegionManager().getRootRegionLocation(),
HRegionInfo.ROOT_REGIONINFO);
} else {
this.metaRegion =
master.getRegionManager().getFirstMetaRegionForRegion(regionInfo);
if (this.metaRegion != null) {
this.metaRegionName = this.metaRegion.getRegionName();
}
}
return this.metaRegion;
}
public HRegionInfo getRegionInfo() {
return regionInfo;
}
}

379
src/main/java/org/apache/hadoop/hbase/master/ProcessServerShutdown.java
View File

@ -1,379 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.wal.HLog;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.master.RegionManager.RegionState;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* Instantiated when a server's lease has expired, meaning it has crashed.
* The region server's log file needs to be split up for each region it was
* serving, and the regions need to get reassigned.
*/
class ProcessServerShutdown extends RegionServerOperation {
// Server name made of the concatenation of hostname, port and startcode
// formatted as <code>&lt;hostname> ',' &lt;port> ',' &lt;startcode></code>
private final String deadServer;
private boolean isRootServer;
private List<MetaRegion> metaRegions;
private Path rsLogDir;
private boolean logSplit;
private boolean rootRescanned;
private HServerAddress deadServerAddress;
private static class ToDoEntry {
boolean regionOffline;
final HRegionInfo info;
ToDoEntry(final HRegionInfo info) {
this.regionOffline = false;
this.info = info;
}
}
/**
* @param master
* @param serverInfo
*/
public ProcessServerShutdown(HMaster master, HServerInfo serverInfo) {
super(master);
this.deadServer = serverInfo.getServerName();
this.deadServerAddress = serverInfo.getServerAddress();
this.logSplit = false;
this.rootRescanned = false;
this.rsLogDir =
new Path(master.getRootDir(), HLog.getHLogDirectoryName(serverInfo));
// check to see if I am responsible for either ROOT or any of the META tables.
// TODO Why do we do this now instead of at processing time?
closeMetaRegions();
}
private void closeMetaRegions() {
this.isRootServer =
this.master.getRegionManager().isRootServer(this.deadServerAddress) ||
this.master.getRegionManager().isRootInTransitionOnThisServer(deadServer);
if (this.isRootServer) {
this.master.getRegionManager().unsetRootRegion();
}
List<byte[]> metaStarts =
this.master.getRegionManager().listMetaRegionsForServer(deadServerAddress);
this.metaRegions = new ArrayList<MetaRegion>();
for (byte [] startKey: metaStarts) {
MetaRegion r = master.getRegionManager().offlineMetaRegionWithStartKey(startKey);
this.metaRegions.add(r);
}
//HBASE-1928: Check whether this server has been transitioning the META table
HRegionInfo metaServerRegionInfo = master.getRegionManager().getMetaServerRegionInfo (deadServer);
if (metaServerRegionInfo != null) {
metaRegions.add (new MetaRegion (deadServerAddress, metaServerRegionInfo));
}
}
/**
* @return Name of server we are processing.
*/
public HServerAddress getDeadServerAddress() {
return this.deadServerAddress;
}
private void closeRegionsInTransition() {
Map<String, RegionState> inTransition =
master.getRegionManager().getRegionsInTransitionOnServer(deadServer);
for (Map.Entry<String, RegionState> entry : inTransition.entrySet()) {
String regionName = entry.getKey();
RegionState state = entry.getValue();
LOG.info("Region " + regionName + " was in transition " +
state + " on dead server " + deadServer + " - marking unassigned");
master.getRegionManager().setUnassigned(state.getRegionInfo(), true);
}
}
@Override
public String toString() {
return "ProcessServerShutdown of " + this.deadServer;
}
/** Finds regions that the dead region server was serving
*/
protected void scanMetaRegion(HRegionInterface server, long scannerId,
byte [] regionName)
throws IOException {
List<ToDoEntry> toDoList = new ArrayList<ToDoEntry>();
Set<HRegionInfo> regions = new HashSet<HRegionInfo>();
List<byte []> emptyRows = new ArrayList<byte []>();
try {
while (true) {
Result values = null;
try {
values = server.next(scannerId);
} catch (IOException e) {
LOG.error("Shutdown scanning of meta region",
RemoteExceptionHandler.checkIOException(e));
break;
}
if (values == null || values.size() == 0) {
break;
}
byte [] row = values.getRow();
// Check server name. If null, skip (We used to consider it was on
// shutdown server but that would mean that we'd reassign regions that
// were already out being assigned, ones that were product of a split
// that happened while the shutdown was being processed).
String serverAddress = BaseScanner.getServerAddress(values);
long startCode = BaseScanner.getStartCode(values);
String serverName = null;
if (serverAddress != null && serverAddress.length() > 0) {
serverName = HServerInfo.getServerName(serverAddress, startCode);
}
if (serverName == null || !deadServer.equals(serverName)) {
// This isn't the server you're looking for - move along
continue;
}
if (LOG.isDebugEnabled() && row != null) {
LOG.debug("Shutdown scanner for " + serverName + " processing " +
Bytes.toString(row));
}
HRegionInfo info = master.getHRegionInfo(row, values);
if (info == null) {
emptyRows.add(row);
continue;
}
synchronized (master.getRegionManager()) {
if (info.isMetaTable()) {
if (LOG.isDebugEnabled()) {
LOG.debug("removing meta region " +
Bytes.toString(info.getRegionName()) +
" from online meta regions");
}
master.getRegionManager().offlineMetaRegionWithStartKey(info.getStartKey());
}
ToDoEntry todo = new ToDoEntry(info);
toDoList.add(todo);
if (master.getRegionManager().isOfflined(info.getRegionNameAsString()) ||
info.isOffline()) {
master.getRegionManager().removeRegion(info);
// Mark region offline
if (!info.isOffline()) {
todo.regionOffline = true;
}
} else {
if (!info.isOffline() && !info.isSplit()) {
// Get region reassigned
regions.add(info);
}
}
}
}
} finally {
if (scannerId != -1L) {
try {
server.close(scannerId);
} catch (IOException e) {
LOG.error("Closing scanner",
RemoteExceptionHandler.checkIOException(e));
}
}
}
// Scan complete. Remove any rows which had empty HRegionInfos
if (emptyRows.size() > 0) {
LOG.warn("Found " + emptyRows.size() +
" rows with empty HRegionInfo while scanning meta region " +
Bytes.toString(regionName));
master.deleteEmptyMetaRows(server, regionName, emptyRows);
}
// Update server in root/meta entries
for (ToDoEntry e: toDoList) {
if (e.regionOffline) {
HRegion.offlineRegionInMETA(server, regionName, e.info);
}
}
// Get regions reassigned
for (HRegionInfo info: regions) {
master.getRegionManager().setUnassigned(info, true);
}
}
private class ScanRootRegion extends RetryableMetaOperation<Boolean> {
ScanRootRegion(MetaRegion m, HMaster master) {
super(m, master);
}
public Boolean call() throws IOException {
if (LOG.isDebugEnabled()) {
LOG.debug("Process server shutdown scanning root region on " +
master.getRegionManager().getRootRegionLocation().getBindAddress());
}
Scan scan = new Scan();
scan.addFamily(HConstants.CATALOG_FAMILY);
long scannerId = server.openScanner(
HRegionInfo.ROOT_REGIONINFO.getRegionName(), scan);
scanMetaRegion(server, scannerId,
HRegionInfo.ROOT_REGIONINFO.getRegionName());
return true;
}
}
private class ScanMetaRegions extends RetryableMetaOperation<Boolean> {
ScanMetaRegions(MetaRegion m, HMaster master) {
super(m, master);
}
public Boolean call() throws IOException {
if (LOG.isDebugEnabled()) {
LOG.debug("process server shutdown scanning " +
Bytes.toString(m.getRegionName()) + " on " + m.getServer());
}
Scan scan = new Scan();
scan.addFamily(HConstants.CATALOG_FAMILY);
long scannerId = server.openScanner(
m.getRegionName(), scan);
scanMetaRegion(server, scannerId, m.getRegionName());
return true;
}
}
@Override
protected boolean process() throws IOException {
LOG.info("Process shutdown of server " + this.deadServer +
": logSplit: " + logSplit + ", rootRescanned: " + rootRescanned +
", numberOfMetaRegions: " + master.getRegionManager().numMetaRegions() +
", onlineMetaRegions.size(): " +
master.getRegionManager().numOnlineMetaRegions());
if (!logSplit) {
// Process the old log file
if (this.master.getFileSystem().exists(rsLogDir)) {
if (!master.splitLogLock.tryLock()) {
return false;
}
try {
HLog.splitLog(master.getRootDir(), rsLogDir,
this.master.getOldLogDir(), this.master.getFileSystem(),
this.master.getConfiguration());
} finally {
master.splitLogLock.unlock();
}
}
logSplit = true;
}
LOG.info("Log split complete, meta reassignment and scanning:");
if (this.isRootServer) {
LOG.info("ProcessServerShutdown reassigning ROOT region");
master.getRegionManager().reassignRootRegion();
isRootServer = false; // prevent double reassignment... heh.
}
for (MetaRegion metaRegion : metaRegions) {
LOG.info("ProcessServerShutdown setting to unassigned: " + metaRegion.toString());
master.getRegionManager().setUnassigned(metaRegion.getRegionInfo(), true);
}
// one the meta regions are online, "forget" about them. Since there are explicit
// checks below to make sure meta/root are online, this is likely to occur.
metaRegions.clear();
if (!rootAvailable()) {
// Return true so that worker does not put this request back on the
// toDoQueue.
// rootAvailable() has already put it on the delayedToDoQueue
return true;
}
if (!rootRescanned) {
// Scan the ROOT region
Boolean result = new ScanRootRegion(
new MetaRegion(master.getRegionManager().getRootRegionLocation(),
HRegionInfo.ROOT_REGIONINFO), this.master).doWithRetries();
if (result == null) {
// Master is closing - give up
return true;
}
if (LOG.isDebugEnabled()) {
LOG.debug("Process server shutdown scanning root region on " +
master.getRegionManager().getRootRegionLocation().getBindAddress() +
" finished " + Thread.currentThread().getName());
}
rootRescanned = true;
}
if (!metaTableAvailable()) {
// We can't proceed because not all meta regions are online.
// metaAvailable() has put this request on the delayedToDoQueue
// Return true so that worker does not put this on the toDoQueue
return true;
}
List<MetaRegion> regions = master.getRegionManager().getListOfOnlineMetaRegions();
for (MetaRegion r: regions) {
Boolean result = new ScanMetaRegions(r, this.master).doWithRetries();
if (result == null) {
break;
}
if (LOG.isDebugEnabled()) {
LOG.debug("process server shutdown finished scanning " +
Bytes.toString(r.getRegionName()) + " on " + r.getServer());
}
}
closeRegionsInTransition();
this.master.getServerManager().removeDeadServer(deadServer);
if (LOG.isDebugEnabled()) {
LOG.debug("Removed " + deadServer + " from deadservers Map");
}
return true;
}
@Override
protected int getPriority() {
return 2; // high but not highest priority
}
}

1690
src/main/java/org/apache/hadoop/hbase/master/RegionManager.java
View File

File diff suppressed because it is too large Load Diff

123
src/main/java/org/apache/hadoop/hbase/master/RegionServerOperation.java
View File

@ -1,123 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import java.io.IOException;
import java.util.concurrent.Delayed;
import java.util.concurrent.TimeUnit;
abstract class RegionServerOperation implements Delayed {
protected static final Log LOG =
LogFactory.getLog(RegionServerOperation.class.getName());
private long expire;
protected final HMaster master;
/* How long we stay on queue.
*/
private int delay;
protected RegionServerOperation(HMaster master) {
this.master = master;
this.delay = this.master.getConfiguration().
getInt("hbase.server.thread.wakefrequency", 10 * 1000);
// Set the future time at which we expect to be released from the
// DelayQueue we're inserted in on lease expiration.
resetExpiration();
}
/**
* Call before putting this back on the delay queue.
* @return When we will expire next.
*/
long resetExpiration() {
// Set the future time at which we expect to be released from the
// DelayQueue we're inserted in on lease expiration.
this.expire = System.currentTimeMillis() + this.delay;
return this.expire;
}
public long getDelay(TimeUnit unit) {
return unit.convert(this.expire - System.currentTimeMillis(),
TimeUnit.MILLISECONDS);
}
void setDelay(final int d) {
this.delay = d;
}
public int compareTo(Delayed o) {
return Long.valueOf(getDelay(TimeUnit.MILLISECONDS)
- o.getDelay(TimeUnit.MILLISECONDS)).intValue();
}
protected void requeue() {
this.master.getRegionServerOperationQueue().putOnDelayQueue(this);
}
private long whenToExpire() {
return System.currentTimeMillis() + this.delay;
}
protected boolean rootAvailable() {
boolean available = true;
if (this.master.getRegionManager().getRootRegionLocation() == null) {
available = false;
requeue();
}
return available;
}
protected boolean metaTableAvailable() {
boolean available = true;
if ((master.getRegionManager().numMetaRegions() !=
master.getRegionManager().numOnlineMetaRegions()) ||
master.getRegionManager().metaRegionsInTransition()) {
// We can't proceed because not all of the meta regions are online.
// We can't block either because that would prevent the meta region
// online message from being processed. In order to prevent spinning
// in the run queue, put this request on the delay queue to give
// other threads the opportunity to get the meta regions on-line.
if (LOG.isDebugEnabled()) {
LOG.debug("numberOfMetaRegions: " +
master.getRegionManager().numMetaRegions() +
", onlineMetaRegions.size(): " +
master.getRegionManager().numOnlineMetaRegions());
LOG.debug("Requeuing because not all meta regions are online");
}
available = false;
requeue();
}
return available;
}
public int compareTo(RegionServerOperation other) {
return getPriority() - other.getPriority();
}
// the Priority of this operation, 0 is lowest priority
protected int getPriority() {
return Integer.MAX_VALUE;
}
protected abstract boolean process() throws IOException;
}

58
src/main/java/org/apache/hadoop/hbase/master/RegionServerOperationListener.java
View File

@ -1,58 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.io.IOException;
import org.apache.hadoop.hbase.HMsg;
import org.apache.hadoop.hbase.HServerInfo;
/**
* Listener for regionserver events in master.
* @see HMaster#registerRegionServerOperationListener(RegionServerOperationListener)
* @see HMaster#unregisterRegionServerOperationListener(RegionServerOperationListener)
*/
public interface RegionServerOperationListener {
/**
* Called for each message passed the master. Most of the messages that come
* in here will go on to become {@link #process(RegionServerOperation)}s but
* others like {@linke HMsg.Type#MSG_REPORT_PROCESS_OPEN} go no further;
* only in here can you see them come in.
* @param serverInfo Server we got the message from.
* @param incomingMsg The message received.
* @return True to continue processing, false to skip.
*/
public boolean process(final HServerInfo serverInfo,
final HMsg incomingMsg);
/**
* Called before processing <code>op</code>
* @param op
* @return True if we are to proceed w/ processing.
* @exception IOException
*/
public boolean process(final RegionServerOperation op) throws IOException;
/**
* Called after <code>op</code> has been processed.
* @param op The operation that just completed.
*/
public void processed(final RegionServerOperation op);
}

257
src/main/java/org/apache/hadoop/hbase/master/RegionServerOperationQueue.java
View File

@ -1,257 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.io.IOException;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.CopyOnWriteArraySet;
import java.util.concurrent.DelayQueue;
import java.util.concurrent.PriorityBlockingQueue;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HMsg;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.util.Sleeper;
import org.apache.hadoop.ipc.RemoteException;
/**
* Keeps up the queue of {@link RegionServerOperation}s.
* Has both live queue and a temporary put-aside queue; if processing of the
* live todo queue fails for some reason, we'll add the item back on the delay
* queue for retry later. Call {@link #shutdown()} to effect a cleanup of
* queues when done. Listen to this queue by registering
* {@link RegionServerOperationListener}s.
* @see #registerRegionServerOperationListener(RegionServerOperationListener)
* @see #unregisterRegionServerOperationListener(RegionServerOperationListener)
*/
public class RegionServerOperationQueue {
// TODO: Build up the junit test of this class.
private final Log LOG = LogFactory.getLog(this.getClass());
/**
* Enums returned by {@link RegionServerOperationQueue#process()};
*/
public static enum ProcessingResultCode {
/**
* Operation was processed successfully.
*/
PROCESSED,
/**
* Nothing to do.
*/
NOOP,
/**
* Operation was put-aside for now. Will be retried later.
*/
REQUEUED,
/**
* Failed processing of the operation.
*/
FAILED,
/**
* Operation was requeued but we failed its processing for some reason
* (Bad filesystem?).
*/
REQUEUED_BUT_PROBLEM
};
/*
* Do not put items directly on this queue. Use {@link #putOnDelayQueue(RegionServerOperation)}.
* It makes sure the expiration on the RegionServerOperation added is updated.
*/
private final DelayQueue<RegionServerOperation> delayedToDoQueue =
new DelayQueue<RegionServerOperation>();
private final BlockingQueue<RegionServerOperation> toDoQueue =
new PriorityBlockingQueue<RegionServerOperation>();
private final Set<RegionServerOperationListener> listeners =
new CopyOnWriteArraySet<RegionServerOperationListener>();
private final int threadWakeFrequency;
private final AtomicBoolean closed;
private final Sleeper sleeper;
RegionServerOperationQueue(final Configuration c, final AtomicBoolean closed) {
this.threadWakeFrequency = c.getInt(HConstants.THREAD_WAKE_FREQUENCY, 10 * 1000);
this.closed = closed;
this.sleeper = new Sleeper(this.threadWakeFrequency, this.closed);
}
public void put(final RegionServerOperation op) {
try {
this.toDoQueue.put(op);
} catch (InterruptedException e) {
LOG.warn("Insertion into todo queue interrupted; putting on delay queue", e);
putOnDelayQueue(op);
}
}
/**
* Try to get an operation off of the queue and process it.
* @return {@link ProcessingResultCode#PROCESSED},
* {@link ProcessingResultCode#REQUEUED},
* {@link ProcessingResultCode#REQUEUED_BUT_PROBLEM}
*/
public synchronized ProcessingResultCode process() {
RegionServerOperation op = null;
// Only process the delayed queue if root region is online. If offline,
// the operation to put it online is probably in the toDoQueue. Process
// it first.
if (toDoQueue.isEmpty()) {
op = delayedToDoQueue.poll();
}
if (op == null) {
try {
op = toDoQueue.poll(threadWakeFrequency, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
LOG.debug("Interrupted", e);
}
}
// At this point, if there's still no todo operation, or we're supposed to
// be closed, return.
if (op == null || closed.get()) {
return ProcessingResultCode.NOOP;
}
try {
if (LOG.isDebugEnabled()) {
LOG.debug("Processing todo: " + op.toString());
}
if (!process(op)) {
// Add it back on the queue.
putOnDelayQueue(op);
} else if (op.process()) {
processed(op);
} else {
// Operation would have blocked because not all meta regions are
// online. This could cause a deadlock, because this thread is waiting
// for the missing meta region(s) to come back online, but since it
// is waiting, it cannot process the meta region online operation it
// is waiting for. So put this operation back on the queue for now.
if (toDoQueue.size() == 0) {
// The queue is currently empty so wait for a while to see if what
// we need comes in first
this.sleeper.sleep();
}
try {
if (LOG.isDebugEnabled()) {
LOG.debug("Put " + op.toString() + " back on queue");
}
toDoQueue.put(op);
} catch (InterruptedException e) {
throw new RuntimeException(
"Putting into toDoQueue was interrupted.", e);
}
}
} catch (Exception ex) {
// There was an exception performing the operation.
if (ex instanceof RemoteException) {
try {
ex = RemoteExceptionHandler.decodeRemoteException(
(RemoteException)ex);
} catch (IOException e) {
ex = e;
LOG.warn("main processing loop: " + op.toString(), e);
}
}
LOG.warn("Failed processing: " + op.toString() +
"; putting onto delayed todo queue", ex);
putOnDelayQueue(op);
return ProcessingResultCode.REQUEUED_BUT_PROBLEM;
}
return ProcessingResultCode.REQUEUED;
}
void putOnDelayQueue(final RegionServerOperation op) {
op.resetExpiration();
this.delayedToDoQueue.put(op);
}
/**
* Clean up the queues.
*/
public synchronized void shutdown() {
this.toDoQueue.clear();
this.delayedToDoQueue.clear();
}
/**
* @param l Register this listener of RegionServerOperation events.
*/
public void registerRegionServerOperationListener(final RegionServerOperationListener l) {
this.listeners.add(l);
}
/**
* @param l Unregister this listener for RegionServerOperation events.
* @return True if this listener was registered.
*/
public boolean unregisterRegionServerOperationListener(final RegionServerOperationListener l) {
return this.listeners.remove(l);
}
/*
* Tell listeners that we processed a RegionServerOperation.
* @param op Operation to tell the world about.
*/
private void processed(final RegionServerOperation op) {
if (this.listeners.isEmpty()) return;
for (RegionServerOperationListener listener: this.listeners) {
listener.processed(op);
}
}
/**
* Called for each message passed the master. Most of the messages that come
* in here will go on to become {@link #process(RegionServerOperation)}s but
* others like {@linke HMsg.Type#MSG_REPORT_PROCESS_OPEN} go no further;
* only in here can you see them come in.
* @param serverInfo Server we got the message from.
* @param incomingMsg The message received.
* @return True to continue processing, false to skip.
*/
boolean process(final HServerInfo serverInfo,
final HMsg incomingMsg) {
if (this.listeners.isEmpty()) return true;
for (RegionServerOperationListener listener: this.listeners) {
if (!listener.process(serverInfo, incomingMsg)) return false;
}
return true;
}
/*
* Tell listeners that we processed a RegionServerOperation.
* @param op Operation to tell the world about.
*/
private boolean process(final RegionServerOperation op) throws IOException {
if (this.listeners.isEmpty()) return true;
for (RegionServerOperationListener listener: this.listeners) {
if (!listener.process(op)) return false;
}
return true;
}
}

103
src/main/java/org/apache/hadoop/hbase/master/RetryableMetaOperation.java
View File

@ -1,103 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.TableNotDisabledException;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Sleeper;
import org.apache.hadoop.ipc.RemoteException;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
/**
* Uses Callable pattern so that operations against meta regions do not need
* to duplicate retry logic.
*/
abstract class RetryableMetaOperation<T> implements Callable<T> {
protected final Log LOG = LogFactory.getLog(this.getClass());
protected final Sleeper sleeper;
protected final MetaRegion m;
protected final HMaster master;
protected HRegionInterface server;
protected RetryableMetaOperation(MetaRegion m, HMaster master) {
this.m = m;
this.master = master;
this.sleeper = new Sleeper(this.master.getThreadWakeFrequency(),
this.master.getClosed());
}
protected T doWithRetries()
throws IOException, RuntimeException {
List<IOException> exceptions = new ArrayList<IOException>();
for (int tries = 0; tries < this.master.getNumRetries(); tries++) {
if (this.master.isClosed()) {
return null;
}
try {
this.server =
this.master.getServerConnection().getHRegionConnection(m.getServer());
return this.call();
} catch (IOException e) {
if (e instanceof TableNotFoundException ||
e instanceof TableNotDisabledException ||
e instanceof InvalidColumnNameException) {
throw e;
}
if (e instanceof RemoteException) {
e = RemoteExceptionHandler.decodeRemoteException((RemoteException) e);
}
if (tries == this.master.getNumRetries() - 1) {
if (LOG.isDebugEnabled()) {
StringBuilder message = new StringBuilder(
"Trying to contact region server for regionName '" +
Bytes.toString(m.getRegionName()) + "', but failed after " +
(tries + 1) + " attempts.\n");
int i = 1;
for (IOException e2 : exceptions) {
message.append("Exception " + i + ":\n" + e2);
}
LOG.debug(message);
}
this.master.checkFileSystem();
throw e;
}
if (LOG.isDebugEnabled()) {
exceptions.add(e);
}
} catch (Exception e) {
LOG.debug("Exception in RetryableMetaOperation: ", e);
throw new RuntimeException(e);
}
this.sleeper.sleep();
}
return null;
}
}

81
src/main/java/org/apache/hadoop/hbase/master/RootScanner.java
View File

@ -1,81 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import java.io.IOException;
/** Scanner for the <code>ROOT</code> HRegion. */
class RootScanner extends BaseScanner {
/**
* Constructor
* @param master
*/
public RootScanner(HMaster master) {
super(master, true, master.getShutdownRequested());
}
/**
* Don't retry if we get an error while scanning. Errors are most often
*
* caused by the server going away. Wait until next rescan interval when
* things should be back to normal.
* @return True if successfully scanned.
*/
private boolean scanRoot() {
master.getRegionManager().waitForRootRegionLocation();
if (master.isClosed()) {
return false;
}
try {
// Don't interrupt us while we're working
synchronized(scannerLock) {
if (master.getRegionManager().getRootRegionLocation() != null) {
scanRegion(new MetaRegion(master.getRegionManager().getRootRegionLocation(),
HRegionInfo.ROOT_REGIONINFO));
}
}
} catch (IOException e) {
e = RemoteExceptionHandler.checkIOException(e);
LOG.warn("Scan ROOT region", e);
// Make sure the file system is still available
master.checkFileSystem();
} catch (Exception e) {
// If for some reason we get some other kind of exception,
// at least log it rather than go out silently.
LOG.error("Unexpected exception", e);
}
return true;
}
@Override
protected boolean initialScan() {
this.initialScanComplete = scanRoot();
return initialScanComplete;
}
@Override
protected void maintenanceScan() {
scanRoot();
}
}

939
src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
View File

File diff suppressed because it is too large Load Diff

77
src/main/java/org/apache/hadoop/hbase/master/TableDelete.java
View File

@ -1,77 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.TableNotDisabledException;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
/**
* Instantiated to delete a table. Table must be offline.
*/
class TableDelete extends TableOperation {
private final Log LOG = LogFactory.getLog(this.getClass());
TableDelete(final HMaster master, final byte [] tableName) throws IOException {
super(master, tableName);
}
@Override
protected void processScanItem(String serverName,
final HRegionInfo info) throws IOException {
if (isEnabled(info)) {
LOG.debug("Region still enabled: " + info.toString());
throw new TableNotDisabledException(tableName);
}
}
@Override
protected void postProcessMeta(MetaRegion m, HRegionInterface server)
throws IOException {
for (HRegionInfo i: unservedRegions) {
if (!Bytes.equals(this.tableName, i.getTableDesc().getName())) {
// Don't delete regions that are not from our table.
continue;
}
// Delete the region
try {
HRegion.removeRegionFromMETA(server, m.getRegionName(), i.getRegionName());
HRegion.deleteRegion(this.master.getFileSystem(),
this.master.getRootDir(), i);
} catch (IOException e) {
LOG.error("failed to delete region " + Bytes.toString(i.getRegionName()),
RemoteExceptionHandler.checkIOException(e));
}
}
// delete the table's folder from fs.
this.master.getFileSystem().delete(new Path(this.master.getRootDir(),
Bytes.toString(this.tableName)), true);
}
}

180
src/main/java/org/apache/hadoop/hbase/master/TableOperation.java
View File

@ -1,180 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.RemoteExceptionHandler;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.TreeSet;
/**
* Abstract base class for operations that need to examine all HRegionInfo
* objects in a table. (For a table, operate on each of its rows
* in .META.).
*/
abstract class TableOperation {
private final Set<MetaRegion> metaRegions;
protected final byte [] tableName;
// Do regions in order.
protected final Set<HRegionInfo> unservedRegions = new TreeSet<HRegionInfo>();
protected HMaster master;
protected TableOperation(final HMaster master, final byte [] tableName)
throws IOException {
this.master = master;
if (!this.master.isMasterRunning()) {
throw new MasterNotRunningException();
}
// add the delimiters.
// TODO maybe check if this is necessary?
this.tableName = tableName;
// Don't wait for META table to come on line if we're enabling it
if (!Bytes.equals(HConstants.META_TABLE_NAME, this.tableName)) {
// We can not access any meta region if they have not already been
// assigned and scanned.
if (master.getRegionManager().metaScannerThread.waitForMetaRegionsOrClose()) {
// We're shutting down. Forget it.
throw new MasterNotRunningException();
}
}
this.metaRegions = master.getRegionManager().getMetaRegionsForTable(tableName);
}
private class ProcessTableOperation extends RetryableMetaOperation<Boolean> {
ProcessTableOperation(MetaRegion m, HMaster master) {
super(m, master);
}
public Boolean call() throws IOException {
boolean tableExists = false;
// Open a scanner on the meta region
byte [] tableNameMetaStart =
Bytes.toBytes(Bytes.toString(tableName) + ",,");
final Scan scan = new Scan(tableNameMetaStart)
.addFamily(HConstants.CATALOG_FAMILY);
long scannerId = this.server.openScanner(m.getRegionName(), scan);
int rows = this.master.getConfiguration().
getInt("hbase.meta.scanner.caching", 100);
scan.setCaching(rows);
List<byte []> emptyRows = new ArrayList<byte []>();
try {
while (true) {
Result values = this.server.next(scannerId);
if (values == null || values.isEmpty()) {
break;
}
HRegionInfo info = this.master.getHRegionInfo(values.getRow(), values);
if (info == null) {
emptyRows.add(values.getRow());
LOG.error(Bytes.toString(HConstants.CATALOG_FAMILY) + ":"
+ Bytes.toString(HConstants.REGIONINFO_QUALIFIER)
+ " not found on "
+ Bytes.toStringBinary(values.getRow()));
continue;
}
final String serverAddress = BaseScanner.getServerAddress(values);
String serverName = null;
if (serverAddress != null && serverAddress.length() > 0) {
long startCode = BaseScanner.getStartCode(values);
serverName = HServerInfo.getServerName(serverAddress, startCode);
}
if (Bytes.compareTo(info.getTableDesc().getName(), tableName) > 0) {
break; // Beyond any more entries for this table
}
tableExists = true;
if (!isBeingServed(serverName) || !isEnabled(info)) {
unservedRegions.add(info);
}
processScanItem(serverName, info);
}
} finally {
if (scannerId != -1L) {
try {
this.server.close(scannerId);
} catch (IOException e) {
e = RemoteExceptionHandler.checkIOException(e);
LOG.error("closing scanner", e);
}
}
scannerId = -1L;
}
// Get rid of any rows that have a null HRegionInfo
if (emptyRows.size() > 0) {
LOG.warn("Found " + emptyRows.size() +
" rows with empty HRegionInfo while scanning meta region " +
Bytes.toString(m.getRegionName()));
master.deleteEmptyMetaRows(server, m.getRegionName(), emptyRows);
}
if (!tableExists) {
throw new TableNotFoundException(Bytes.toString(tableName));
}
postProcessMeta(m, server);
unservedRegions.clear();
return Boolean.TRUE;
}
}
void process() throws IOException {
// Prevent meta scanner from running
synchronized(master.getRegionManager().metaScannerThread.scannerLock) {
for (MetaRegion m: metaRegions) {
new ProcessTableOperation(m, master).doWithRetries();
}
}
}
protected boolean isBeingServed(String serverName) {
boolean result = false;
if (serverName != null && serverName.length() > 0) {
HServerInfo s = master.getServerManager().getServerInfo(serverName);
result = s != null;
}
return result;
}
protected boolean isEnabled(HRegionInfo info) {
return !info.isOffline();
}
protected abstract void processScanItem(String serverName, HRegionInfo info)
throws IOException;
protected abstract void postProcessMeta(MetaRegion m,
HRegionInterface server) throws IOException;
}

129
src/main/java/org/apache/hadoop/hbase/master/ZKMasterAddressWatcher.java
View File

@ -1,129 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.Watcher.Event.EventType;
import java.util.concurrent.atomic.AtomicBoolean;
/**
* ZooKeeper watcher for the master address. Also watches the cluster state
* flag so will shutdown this master if cluster has been shutdown.
* <p>Used by the Master. Waits on the master address ZNode delete event. When
* multiple masters are brought up, they race to become master by writing their
* address to ZooKeeper. Whoever wins becomes the master, and the rest wait for
* that ephemeral node in ZooKeeper to evaporate (meaning the master went down),
* at which point they try to write their own address to become the new master.
*/
class ZKMasterAddressWatcher implements Watcher {
private static final Log LOG = LogFactory.getLog(ZKMasterAddressWatcher.class);
private ZooKeeperWrapper zookeeper;
private final AtomicBoolean requestShutdown;
/**
* Create this watcher using passed ZooKeeperWrapper instance.
* @param zk ZooKeeper
* @param flag Flag to set to request shutdown.
*/
ZKMasterAddressWatcher(final ZooKeeperWrapper zk, final AtomicBoolean flag) {
this.requestShutdown = flag;
this.zookeeper = zk;
}
/**
* @see org.apache.zookeeper.Watcher#process(org.apache.zookeeper.WatchedEvent)
*/
@Override
public synchronized void process (WatchedEvent event) {
EventType type = event.getType();
LOG.debug(("Got event " + type + " with path " + event.getPath()));
if (type.equals(EventType.NodeDeleted)) {
if (event.getPath().equals(this.zookeeper.clusterStateZNode)) {
LOG.info("Cluster shutdown while waiting, shutting down" +
" this master.");
this.requestShutdown.set(true);
} else {
LOG.debug("Master address ZNode deleted, notifying waiting masters");
notifyAll();
}
} else if(type.equals(EventType.NodeCreated) &&
event.getPath().equals(this.zookeeper.clusterStateZNode)) {
LOG.debug("Resetting watch on cluster state node.");
this.zookeeper.setClusterStateWatch();
}
}
/**
* Wait for master address to be available. This sets a watch in ZooKeeper and
* blocks until the master address ZNode gets deleted.
*/
public synchronized void waitForMasterAddressAvailability() {
while (zookeeper.readMasterAddress(this) != null) {
try {
LOG.debug("Waiting for master address ZNode to be deleted " +
"(Also watching cluster state node)");
this.zookeeper.setClusterStateWatch();
wait();
} catch (InterruptedException e) {
}
}
}
/**
* Write address to zookeeper. Parks here until we successfully write our
* address (or until cluster shutdown).
* @param address Address whose format is HServerAddress.toString
*/
boolean writeAddressToZooKeeper(
final HServerAddress address, boolean retry) {
do {
waitForMasterAddressAvailability();
// Check if we need to shutdown instead of taking control
if (this.requestShutdown.get()) {
LOG.debug("Won't start Master because cluster is shuting down");
return false;
}
if(this.zookeeper.writeMasterAddress(address)) {
this.zookeeper.setClusterState(true);
this.zookeeper.setClusterStateWatch();
// Watch our own node
this.zookeeper.readMasterAddress(this);
return true;
}
} while(retry);
return false;
}
/**
* Reset the ZK in case a new connection is required
* @param zookeeper new instance
*/
public void setZookeeper(ZooKeeperWrapper zookeeper) {
this.zookeeper = zookeeper;
}
}

185
src/main/java/org/apache/hadoop/hbase/master/ZKUnassignedWatcher.java
View File

@ -1,185 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import java.io.IOException;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.executor.HBaseEventHandler.HBaseEventType;
import org.apache.hadoop.hbase.master.handler.MasterCloseRegionHandler;
import org.apache.hadoop.hbase.master.handler.MasterOpenRegionHandler;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper;
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ZNodePathAndData;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.Watcher.Event.EventType;
/**
* Watches the UNASSIGNED znode in ZK for the master, and handles all events
* relating to region transitions.
*/
public class ZKUnassignedWatcher implements Watcher {
private static final Log LOG = LogFactory.getLog(ZKUnassignedWatcher.class);
private ZooKeeperWrapper zkWrapper;
String serverName;
ServerManager serverManager;
public static void start(Configuration conf, HMaster master)
throws IOException {
new ZKUnassignedWatcher(conf, master);
LOG.debug("Started ZKUnassigned watcher");
}
public ZKUnassignedWatcher(Configuration conf, HMaster master)
throws IOException {
this.serverName = master.getHServerAddress().toString();
this.serverManager = master.getServerManager();
zkWrapper = ZooKeeperWrapper.getInstance(conf, HMaster.class.getName());
String unassignedZNode = zkWrapper.getRegionInTransitionZNode();
// If the UNASSIGNED ZNode exists and this is a fresh cluster start, then
// delete it.
if(master.isClusterStartup() && zkWrapper.exists(unassignedZNode, false)) {
LOG.info("Cluster start, but found " + unassignedZNode + ", deleting it.");
try {
zkWrapper.deleteZNode(unassignedZNode, true);
} catch (KeeperException e) {
LOG.error("Could not delete znode " + unassignedZNode, e);
throw new IOException(e);
} catch (InterruptedException e) {
LOG.error("Could not delete znode " + unassignedZNode, e);
throw new IOException(e);
}
}
// If the UNASSIGNED ZNode does not exist, create it.
zkWrapper.createZNodeIfNotExists(unassignedZNode);
// TODO: get the outstanding changes in UNASSIGNED
// Set a watch on Zookeeper's UNASSIGNED node if it exists.
zkWrapper.registerListener(this);
}
/**
* This is the processing loop that gets triggered from the ZooKeeperWrapper.
* This zookeeper events process function dies the following:
* - WATCHES the following events: NodeCreated, NodeDataChanged, NodeChildrenChanged
* - IGNORES the following events: None, NodeDeleted
*/
@Override
public synchronized void process(WatchedEvent event) {
EventType type = event.getType();
LOG.debug("ZK-EVENT-PROCESS: Got zkEvent " + type +
" state:" + event.getState() +
" path:" + event.getPath());
// Handle the ignored events
if(type.equals(EventType.None) ||
type.equals(EventType.NodeDeleted)) {
return;
}
// check if the path is for the UNASSIGNED directory we care about
if(event.getPath() == null ||
!event.getPath().startsWith(zkWrapper.getZNodePathForHBase(
zkWrapper.getRegionInTransitionZNode()))) {
return;
}
try
{
/*
* If a node is created in the UNASSIGNED directory in zookeeper, then:
* 1. watch its updates (this is an unassigned region).
* 2. read to see what its state is and handle as needed (state may have
* changed before we started watching it)
*/
if(type.equals(EventType.NodeCreated)) {
zkWrapper.watchZNode(event.getPath());
handleRegionStateInZK(event.getPath());
}
/*
* Data on some node has changed. Read to see what the state is and handle
* as needed.
*/
else if(type.equals(EventType.NodeDataChanged)) {
handleRegionStateInZK(event.getPath());
}
/*
* If there were some nodes created then watch those nodes
*/
else if(type.equals(EventType.NodeChildrenChanged)) {
List<ZNodePathAndData> newZNodes =
zkWrapper.watchAndGetNewChildren(event.getPath());
for(ZNodePathAndData zNodePathAndData : newZNodes) {
LOG.debug("Handling updates for znode: " + zNodePathAndData.getzNodePath());
handleRegionStateInZK(zNodePathAndData.getzNodePath(),
zNodePathAndData.getData());
}
}
}
catch (IOException e)
{
LOG.error("Could not process event from ZooKeeper", e);
}
}
/**
* Read the state of a node in ZK, and do the needful. We want to do the
* following:
* 1. If region's state is updated as CLOSED, invoke the ClosedRegionHandler.
* 2. If region's state is updated as OPENED, invoke the OpenRegionHandler.
* @param zNodePath
* @throws IOException
*/
private void handleRegionStateInZK(String zNodePath) throws IOException {
byte[] data = zkWrapper.readZNode(zNodePath, null);
handleRegionStateInZK(zNodePath, data);
}
private void handleRegionStateInZK(String zNodePath, byte[] data) {
// a null value is set when a node is created, we don't need to handle this
if(data == null) {
return;
}
String rgnInTransitNode = zkWrapper.getRegionInTransitionZNode();
String region = zNodePath.substring(
zNodePath.indexOf(rgnInTransitNode) + rgnInTransitNode.length() + 1);
HBaseEventType rsEvent = HBaseEventType.fromByte(data[0]);
LOG.debug("Got event type [ " + rsEvent + " ] for region " + region);
// if the node was CLOSED then handle it
if(rsEvent == HBaseEventType.RS2ZK_REGION_CLOSED) {
new MasterCloseRegionHandler(rsEvent, serverManager, serverName, region, data).submit();
}
// if the region was OPENED then handle that
else if(rsEvent == HBaseEventType.RS2ZK_REGION_OPENED ||
rsEvent == HBaseEventType.RS2ZK_REGION_OPENING) {
new MasterOpenRegionHandler(rsEvent, serverManager, serverName, region, data).submit();
}
}
}

114
src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java Normal file
View File

@ -0,0 +1,114 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.executor.EventHandler;
import org.apache.hadoop.hbase.executor.RegionTransitionData;
import org.apache.hadoop.hbase.master.AssignmentManager;
import org.apache.hadoop.hbase.zookeeper.ZKAssign;
import org.apache.zookeeper.KeeperException;
/**
* Handles CLOSED region event on Master.
* <p>
* If table is being disabled, deletes ZK unassigned node and removes from
* regions in transition.
* <p>
* Otherwise, assigns the region to another server.
*/
public class ClosedRegionHandler extends EventHandler implements TotesHRegionInfo {
private static final Log LOG = LogFactory.getLog(ClosedRegionHandler.class);
private final AssignmentManager assignmentManager;
private final RegionTransitionData data;
private final HRegionInfo regionInfo;
private final ClosedPriority priority;
private enum ClosedPriority {
ROOT (1),
META (2),
USER (3);
private final int value;
ClosedPriority(int value) {
this.value = value;
}
public int getValue() {
return value;
}
};
public ClosedRegionHandler(Server server,
AssignmentManager assignmentManager, RegionTransitionData data,
HRegionInfo regionInfo) {
super(server, EventType.RS2ZK_REGION_CLOSED);
this.assignmentManager = assignmentManager;
this.data = data;
this.regionInfo = regionInfo;
if(regionInfo.isRootRegion()) {
priority = ClosedPriority.ROOT;
} else if(regionInfo.isMetaRegion()) {
priority = ClosedPriority.META;
} else {
priority = ClosedPriority.USER;
}
}
@Override
public int getPriority() {
return priority.getValue();
}
@Override
public HRegionInfo getHRegionInfo() {
return this.regionInfo;
}
@Override
public void process() {
LOG.debug("Handling CLOSED event");
// Check if this table is being disabled or not
if (assignmentManager.isTableOfRegionDisabled(regionInfo.getRegionName())) {
// Disabling so should not be reassigned, just delete the CLOSED node
LOG.debug("Table being disabled so deleting ZK node and removing from " +
"regions in transition, skipping assignment");
try {
ZKAssign.deleteClosedNode(server.getZooKeeper(),
regionInfo.getEncodedName());
} catch (KeeperException.NoNodeException nne) {
LOG.warn("Tried to delete closed node for " + data + " but it does " +
"not exist");
return;
} catch (KeeperException e) {
server.abort("Error deleting CLOSED node in ZK", e);
}
assignmentManager.regionOffline(regionInfo);
return;
}
// ZK Node is in CLOSED state, assign it.
assignmentManager.setOffline(regionInfo);
assignmentManager.assign(regionInfo);
}
}

53
src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java Normal file
View File

@ -0,0 +1,53 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.catalog.MetaEditor;
import org.apache.hadoop.hbase.master.MasterServices;
public class DeleteTableHandler extends TableEventHandler {
private static final Log LOG = LogFactory.getLog(DeleteTableHandler.class);
public DeleteTableHandler(byte [] tableName, Server server,
final MasterServices masterServices) throws IOException {
super(EventType.C2M_DELETE_TABLE, tableName, server, masterServices);
}
@Override
protected void handleTableOperation(List<HRegionInfo> regions)
throws IOException {
for(HRegionInfo region : regions) {
LOG.debug("Deleting region " + region + " from META and FS");
// Remove region from META
MetaEditor.deleteRegion(this.server.getCatalogTracker(), region);
// Delete region from FS
this.masterServices.getMasterFileSystem().deleteRegion(region);
}
// Delete table from FS
this.masterServices.getMasterFileSystem().deleteTable(tableName);
}
}

83
src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java Normal file
View File

@ -0,0 +1,83 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.catalog.CatalogTracker;
import org.apache.hadoop.hbase.catalog.MetaReader;
import org.apache.hadoop.hbase.executor.EventHandler;
import org.apache.hadoop.hbase.master.AssignmentManager;
import org.apache.hadoop.hbase.util.Bytes;
public class DisableTableHandler extends EventHandler {
private static final Log LOG = LogFactory.getLog(DisableTableHandler.class);
private final byte [] tableName;
private final String tableNameStr;
private final AssignmentManager assignmentManager;
public DisableTableHandler(Server server, byte [] tableName,
CatalogTracker catalogTracker, AssignmentManager assignmentManager)
throws TableNotFoundException, IOException {
super(server, EventType.C2M_DISABLE_TABLE);
this.tableName = tableName;
this.tableNameStr = Bytes.toString(this.tableName);
this.assignmentManager = assignmentManager;
// Check if table exists
// TODO: do we want to keep this in-memory as well? i guess this is
// part of old master rewrite, schema to zk to check for table
// existence and such
if(!MetaReader.tableExists(catalogTracker, this.tableNameStr)) {
throw new TableNotFoundException(Bytes.toString(tableName));
}
}
@Override
public void process() {
try {
LOG.info("Attemping to disable the table " + this.tableNameStr);
handleDisableTable();
} catch (IOException e) {
LOG.error("Error trying to disable the table " + this.tableNameStr, e);
}
}
private void handleDisableTable() throws IOException {
// Set the table as disabled so it doesn't get re-onlined
assignmentManager.disableTable(this.tableNameStr);
// Get the online regions of this table.
// TODO: What if region splitting at the time we get this listing?
// TODO: Remove offline flag from HRI
// TODO: Confirm we have parallel closing going on.
List<HRegionInfo> regions = assignmentManager.getRegionsOfTable(tableName);
// Unassign the online regions
for(HRegionInfo region : regions) {
assignmentManager.unassign(region);
}
}
}

79
src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java Normal file
View File

@ -0,0 +1,79 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.catalog.CatalogTracker;
import org.apache.hadoop.hbase.catalog.MetaReader;
import org.apache.hadoop.hbase.executor.EventHandler;
import org.apache.hadoop.hbase.master.AssignmentManager;
import org.apache.hadoop.hbase.util.Bytes;
public class EnableTableHandler extends EventHandler {
private static final Log LOG = LogFactory.getLog(EnableTableHandler.class);
private final byte [] tableName;
private final String tableNameStr;
private final AssignmentManager assignmentManager;
private final CatalogTracker ct;
public EnableTableHandler(Server server, byte [] tableName,
CatalogTracker catalogTracker, AssignmentManager assignmentManager)
throws TableNotFoundException, IOException {
super(server, EventType.C2M_ENABLE_TABLE);
this.tableName = tableName;
this.tableNameStr = Bytes.toString(tableName);
this.ct = catalogTracker;
this.assignmentManager = assignmentManager;
// Check if table exists
if(!MetaReader.tableExists(catalogTracker, this.tableNameStr)) {
throw new TableNotFoundException(Bytes.toString(tableName));
}
}
@Override
public void process() {
try {
LOG.info("Attemping to enable the table " + this.tableNameStr);
handleEnableTable();
} catch (IOException e) {
LOG.error("Error trying to enable the table " + this.tableNameStr, e);
}
}
private void handleEnableTable() throws IOException {
// Get the regions of this table
List<HRegionInfo> regions = MetaReader.getTableRegions(this.ct, tableName);
// Set the table as disabled so it doesn't get re-onlined
assignmentManager.undisableTable(this.tableNameStr);
// Verify all regions of table are disabled
for (HRegionInfo region : regions) {
assignmentManager.assign(region);
}
}
}

94
src/main/java/org/apache/hadoop/hbase/master/handler/MasterCloseRegionHandler.java
View File

@ -1,94 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.executor.RegionTransitionEventData;
import org.apache.hadoop.hbase.executor.HBaseEventHandler;
import org.apache.hadoop.hbase.master.HMaster;
import org.apache.hadoop.hbase.master.ServerManager;
import org.apache.hadoop.hbase.util.Writables;
/**
* This is the event handler for all events relating to closing regions on the
* HMaster. The following event types map to this handler:
* - RS_REGION_CLOSING
* - RS_REGION_CLOSED
*/
public class MasterCloseRegionHandler extends HBaseEventHandler
{
private static final Log LOG = LogFactory.getLog(MasterCloseRegionHandler.class);
private String regionName;
protected byte[] serializedData;
RegionTransitionEventData hbEventData;
ServerManager serverManager;
public MasterCloseRegionHandler(HBaseEventType eventType,
ServerManager serverManager,
String serverName,
String regionName,
byte[] serializedData) {
super(false, serverName, eventType);
this.regionName = regionName;
this.serializedData = serializedData;
this.serverManager = serverManager;
}
/**
* Handle the various events relating to closing regions. We can get the
* following events here:
* - RS_REGION_CLOSING : No-op
* - RS_REGION_CLOSED : The region is closed. If we are not in a shutdown
* state, find the RS to open this region. This could
* be a part of a region move, or just that the RS has
* died. Should result in a M_REQUEST_OPENREGION event
* getting created.
*/
@Override
public void process()
{
LOG.debug("Event = " + getHBEvent() + ", region = " + regionName);
// handle RS_REGION_CLOSED events
handleRegionClosedEvent();
}
private void handleRegionClosedEvent() {
try {
if(hbEventData == null) {
hbEventData = new RegionTransitionEventData();
Writables.getWritable(serializedData, hbEventData);
}
} catch (IOException e) {
LOG.error("Could not deserialize additional args for Close region", e);
}
// process the region close - this will cause the reopening of the
// region as a part of the heartbeat of some RS
serverManager.processRegionClose(hbEventData.getHmsg().getRegionInfo());
LOG.info("Processed close of region " + hbEventData.getHmsg().getRegionInfo().getRegionNameAsString());
}
public String getRegionName() {
return regionName;
}
}

112
src/main/java/org/apache/hadoop/hbase/master/handler/MasterOpenRegionHandler.java
View File

@ -1,112 +0,0 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.ArrayList;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HMsg;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.executor.RegionTransitionEventData;
import org.apache.hadoop.hbase.executor.HBaseEventHandler;
import org.apache.hadoop.hbase.master.HMaster;
import org.apache.hadoop.hbase.master.ServerManager;
import org.apache.hadoop.hbase.util.Writables;
/**
* This is the event handler for all events relating to opening regions on the
* HMaster. This could be one of the following:
* - notification that a region server is "OPENING" a region
* - notification that a region server has "OPENED" a region
* The following event types map to this handler:
* - RS_REGION_OPENING
* - RS_REGION_OPENED
*/
public class MasterOpenRegionHandler extends HBaseEventHandler {
private static final Log LOG = LogFactory.getLog(MasterOpenRegionHandler.class);
// other args passed in a byte array form
protected byte[] serializedData;
private String regionName;
private RegionTransitionEventData hbEventData;
ServerManager serverManager;
public MasterOpenRegionHandler(HBaseEventType eventType,
ServerManager serverManager,
String serverName,
String regionName,
byte[] serData) {
super(false, serverName, eventType);
this.regionName = regionName;
this.serializedData = serData;
this.serverManager = serverManager;
}
/**
* Handle the various events relating to opening regions. We can get the
* following events here:
* - RS_REGION_OPENING : Keep track to see how long the region open takes.
* If the RS is taking too long, then revert the
* region back to closed state so that it can be
* re-assigned.
* - RS_REGION_OPENED : The region is opened. Add an entry into META for
* the RS having opened this region. Then delete this
* entry in ZK.
*/
@Override
public void process()
{
LOG.debug("Event = " + getHBEvent() + ", region = " + regionName);
if(this.getHBEvent() == HBaseEventType.RS2ZK_REGION_OPENING) {
handleRegionOpeningEvent();
}
else if(this.getHBEvent() == HBaseEventType.RS2ZK_REGION_OPENED) {
handleRegionOpenedEvent();
}
}
private void handleRegionOpeningEvent() {
// TODO: not implemented.
LOG.debug("NO-OP call to handling region opening event");
// Keep track to see how long the region open takes. If the RS is taking too
// long, then revert the region back to closed state so that it can be
// re-assigned.
}
private void handleRegionOpenedEvent() {
try {
if(hbEventData == null) {
hbEventData = new RegionTransitionEventData();
Writables.getWritable(serializedData, hbEventData);
}
} catch (IOException e) {
LOG.error("Could not deserialize additional args for Open region", e);
}
LOG.debug("RS " + hbEventData.getRsName() + " has opened region " + regionName);
HServerInfo serverInfo = serverManager.getServerInfo(hbEventData.getRsName());
ArrayList<HMsg> returnMsgs = new ArrayList<HMsg>();
serverManager.processRegionOpen(serverInfo, hbEventData.getHmsg().getRegionInfo(), returnMsgs);
if(returnMsgs.size() > 0) {
LOG.error("Open region tried to send message: " + returnMsgs.get(0).getType() +
" about " + returnMsgs.get(0).getRegionInfo().getRegionNameAsString());
}
}
}

42
src/main/java/org/apache/hadoop/hbase/master/AddColumn.java → src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java
View File

@ -17,34 +17,36 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.List;
/** Instantiated to add a column family to a table */
class AddColumn extends ColumnOperation {
private final HColumnDescriptor newColumn;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.catalog.MetaEditor;
import org.apache.hadoop.hbase.master.MasterServices;
AddColumn(final HMaster master, final byte [] tableName,
final HColumnDescriptor newColumn)
throws IOException {
super(master, tableName);
this.newColumn = newColumn;
public class ModifyTableHandler extends TableEventHandler {
private final HTableDescriptor htd;
public ModifyTableHandler(final byte [] tableName,
final HTableDescriptor htd, final Server server,
final MasterServices masterServices) throws IOException {
super(EventType.C2M_MODIFY_TABLE, tableName, server, masterServices);
this.htd = htd;
}
@Override
protected void postProcessMeta(MetaRegion m, HRegionInterface server)
protected void handleTableOperation(List<HRegionInfo> hris)
throws IOException {
for (HRegionInfo i: unservedRegions) {
// All we need to do to add a column is add it to the table descriptor.
// When the region is brought on-line, it will find the column missing
// and create it.
i.getTableDesc().addFamily(newColumn);
updateRegionInfo(server, m.getRegionName(), i);
for (HRegionInfo hri : hris) {
// Update region info in META
hri.setTableDesc(this.htd);
MetaEditor.updateRegionInfo(this.server.getCatalogTracker(), hri);
// Update region info in FS
this.masterServices.getMasterFileSystem().updateRegionInfo(hri);
}
}
}

100
src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java Normal file
View File

@ -0,0 +1,100 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.executor.EventHandler;
import org.apache.hadoop.hbase.executor.RegionTransitionData;
import org.apache.hadoop.hbase.master.AssignmentManager;
import org.apache.hadoop.hbase.zookeeper.ZKAssign;
import org.apache.zookeeper.KeeperException;
/**
* Handles OPENED region event on Master.
*/
public class OpenedRegionHandler extends EventHandler implements TotesHRegionInfo {
private static final Log LOG = LogFactory.getLog(OpenedRegionHandler.class);
private final AssignmentManager assignmentManager;
private final RegionTransitionData data;
private final HRegionInfo regionInfo;
private final HServerInfo serverInfo;
private final OpenedPriority priority;
private enum OpenedPriority {
ROOT (1),
META (2),
USER (3);
private final int value;
OpenedPriority(int value) {
this.value = value;
}
public int getValue() {
return value;
}
};
public OpenedRegionHandler(Server server,
AssignmentManager assignmentManager, RegionTransitionData data,
HRegionInfo regionInfo, HServerInfo serverInfo) {
super(server, EventType.RS2ZK_REGION_OPENED);
this.assignmentManager = assignmentManager;
this.data = data;
this.regionInfo = regionInfo;
this.serverInfo = serverInfo;
if(regionInfo.isRootRegion()) {
priority = OpenedPriority.ROOT;
} else if(regionInfo.isMetaRegion()) {
priority = OpenedPriority.META;
} else {
priority = OpenedPriority.USER;
}
}
@Override
public int getPriority() {
return priority.getValue();
}
@Override
public HRegionInfo getHRegionInfo() {
return this.regionInfo;
}
@Override
public void process() {
LOG.debug("Handling OPENED event; deleting unassigned node");
// TODO: should we check if this table was disabled and get it closed?
// Remove region from in-memory transition and unassigned node from ZK
try {
ZKAssign.deleteOpenedNode(server.getZooKeeper(),
regionInfo.getEncodedName());
} catch (KeeperException e) {
server.abort("Error deleting OPENED node in ZK", e);
}
assignmentManager.regionOnline(regionInfo, serverInfo);
LOG.debug("Opened region " + regionInfo.getRegionNameAsString());
}
}

160
src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java Normal file
View File

@ -0,0 +1,160 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.Map;
import java.util.NavigableMap;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HServerInfo;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.catalog.MetaEditor;
import org.apache.hadoop.hbase.catalog.MetaReader;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.executor.EventHandler;
import org.apache.hadoop.hbase.master.DeadServer;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.util.Writables;
import org.apache.zookeeper.KeeperException;
public class ServerShutdownHandler extends EventHandler {
private static final Log LOG = LogFactory.getLog(ServerShutdownHandler.class);
private final HServerInfo hsi;
private final Server server;
private final MasterServices services;
private final DeadServer deadServers;
public ServerShutdownHandler(final Server server, final MasterServices services,
final DeadServer deadServers, final HServerInfo hsi) {
super(server, EventType.M_SERVER_SHUTDOWN);
this.hsi = hsi;
this.server = server;
this.services = services;
this.deadServers = deadServers;
// Add to dead servers.
this.deadServers.add(hsi.getServerName());
}
@Override
public void process() throws IOException {
Pair<Boolean, Boolean> carryingCatalog = null;
try {
carryingCatalog =
this.server.getCatalogTracker().processServerShutdown(this.hsi);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException("Interrupted", e);
} catch (KeeperException e) {
this.server.abort("In server shutdown processing", e);
throw new IOException("Aborting", e);
}
final String serverName = this.hsi.getServerName();
LOG.info("Splitting logs for " + serverName);
this.services.getMasterFileSystem().splitLog(serverName);
// Clean out anything in regions in transition. Being conservative and
// doing after log splitting. Could do some states before -- OPENING?
// OFFLINE? -- and then others after like CLOSING that depend on log
// splitting.
this.services.getAssignmentManager().processServerShutdown(this.hsi);
// Assign root and meta if we were carrying them.
if (carryingCatalog.getFirst()) { // -ROOT-
try {
this.services.getAssignmentManager().assignRoot();
} catch (KeeperException e) {
this.server.abort("In server shutdown processing, assigning root", e);
throw new IOException("Aborting", e);
}
}
if (carryingCatalog.getSecond()) { // .META.
this.services.getAssignmentManager().assignMeta();
}
// Wait on meta to come online; we need it to progress.
try {
this.server.getCatalogTracker().waitForMeta();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException("Interrupted", e);
}
NavigableMap<HRegionInfo, Result> hris =
MetaReader.getServerRegions(this.server.getCatalogTracker(), this.hsi);
LOG.info("Reassigning the " + hris.size() + " region(s) that " + serverName +
" was carrying.");
// We should encounter -ROOT- and .META. first in the Set given how its
// a sorted set.
for (Map.Entry<HRegionInfo, Result> e: hris.entrySet()) {
// If table is not disabled but the region is offlined,
HRegionInfo hri = e.getKey();
boolean disabled = this.services.getAssignmentManager().
isTableDisabled(hri.getTableDesc().getNameAsString());
if (disabled) continue;
if (hri.isOffline() && hri.isSplit()) {
fixupDaughters(hris, e.getValue());
continue;
}
this.services.getAssignmentManager().assign(hri);
}
this.deadServers.remove(serverName);
LOG.info("Finished processing of shutdown of " + serverName);
}
/**
* Check that daughter regions are up in .META. and if not, add them.
* @param hris All regions for this server in meta.
* @param result The contents of the parent row in .META.
* @throws IOException
*/
void fixupDaughters(final NavigableMap<HRegionInfo, Result> hris,
final Result result) throws IOException {
fixupDaughter(hris, result, HConstants.SPLITA_QUALIFIER);
fixupDaughter(hris, result, HConstants.SPLITB_QUALIFIER);
}
/**
* Check individual daughter is up in .META.; fixup if its not.
* @param hris All regions for this server in meta.
* @param result The contents of the parent row in .META.
* @param qualifier Which daughter to check for.
* @throws IOException
*/
void fixupDaughter(final NavigableMap<HRegionInfo, Result> hris,
final Result result, final byte [] qualifier)
throws IOException {
byte [] bytes = result.getValue(HConstants.CATALOG_FAMILY, qualifier);
if (bytes == null || bytes.length <= 0) return;
HRegionInfo hri = Writables.getHRegionInfo(bytes);
if (!hris.containsKey(hri)) {
LOG.info("Fixup; missing daughter " + hri.getEncodedNameAsBytes());
MetaEditor.addDaughter(this.server.getCatalogTracker(), hri, null);
this.services.getAssignmentManager().assign(hri);
}
}
}

66
src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java Normal file
View File

@ -0,0 +1,66 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.List;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.InvalidFamilyOperationException;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.catalog.MetaEditor;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.util.Bytes;
/**
* Handles adding a new family to an existing table.
*/
public class TableAddFamilyHandler extends TableEventHandler {
private final HColumnDescriptor familyDesc;
public TableAddFamilyHandler(byte[] tableName, HColumnDescriptor familyDesc,
Server server, final MasterServices masterServices) throws IOException {
super(EventType.C2M_ADD_FAMILY, tableName, server, masterServices);
this.familyDesc = familyDesc;
}
@Override
protected void handleTableOperation(List<HRegionInfo> hris)
throws IOException {
HTableDescriptor htd = hris.get(0).getTableDesc();
byte [] familyName = familyDesc.getName();
if(htd.hasFamily(familyName)) {
throw new InvalidFamilyOperationException(
"Family '" + Bytes.toString(familyName) + "' already exists so " +
"cannot be added");
}
for(HRegionInfo hri : hris) {
// Update the HTD
hri.getTableDesc().addFamily(familyDesc);
// Update region in META
MetaEditor.updateRegionInfo(this.server.getCatalogTracker(), hri);
// Update region info in FS
this.masterServices.getMasterFileSystem().updateRegionInfo(hri);
}
}
}

69
src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java Normal file
View File

@ -0,0 +1,69 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.List;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.InvalidFamilyOperationException;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.catalog.MetaEditor;
import org.apache.hadoop.hbase.master.MasterFileSystem;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.util.Bytes;
/**
* Handles adding a new family to an existing table.
*/
public class TableDeleteFamilyHandler extends TableEventHandler {
private final byte [] familyName;
public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,
Server server, final MasterServices masterServices) throws IOException {
super(EventType.C2M_ADD_FAMILY, tableName, server, masterServices);
this.familyName = familyName;
}
@Override
protected void handleTableOperation(List<HRegionInfo> hris) throws IOException {
HTableDescriptor htd = hris.get(0).getTableDesc();
if(!htd.hasFamily(familyName)) {
throw new InvalidFamilyOperationException(
"Family '" + Bytes.toString(familyName) + "' does not exist so " +
"cannot be deleted");
}
for (HRegionInfo hri : hris) {
// Update the HTD
hri.getTableDesc().removeFamily(familyName);
// Update region in META
MetaEditor.updateRegionInfo(this.server.getCatalogTracker(), hri);
MasterFileSystem mfs = this.masterServices.getMasterFileSystem();
// Update region info in FS
mfs.updateRegionInfo(hri);
// Delete directory in FS
mfs.deleteFamily(hri, familyName);
// Update region info in FS
this.masterServices.getMasterFileSystem().updateRegionInfo(hri);
}
}
}

72
src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java Normal file
View File

@ -0,0 +1,72 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.catalog.MetaReader;
import org.apache.hadoop.hbase.executor.EventHandler;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.util.Bytes;
/**
* Base class for performing operations against tables.
* Checks on whether the process can go forward are done in constructor rather
* than later on in {@link #process()}. The idea is to fail fast rather than
* later down in an async invocation of {@link #process()} (which currently has
* no means of reporting back issues once started).
*/
public abstract class TableEventHandler extends EventHandler {
private static final Log LOG = LogFactory.getLog(TableEventHandler.class);
protected final MasterServices masterServices;
protected final byte [] tableName;
public TableEventHandler(EventType eventType, byte [] tableName, Server server,
MasterServices masterServices)
throws IOException {
super(server, eventType);
this.masterServices = masterServices;
this.tableName = tableName;
this.masterServices.checkTableModifiable(tableName);
}
@Override
public void process() {
try {
LOG.info("Handling table operation " + eventType + " on table " +
Bytes.toString(tableName));
List<HRegionInfo> hris =
MetaReader.getTableRegions(this.server.getCatalogTracker(),
tableName);
handleTableOperation(hris);
} catch (IOException e) {
LOG.error("Error trying to delete the table " + Bytes.toString(tableName),
e);
}
}
protected abstract void handleTableOperation(List<HRegionInfo> regions)
throws IOException;
}

65
src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java Normal file
View File

@ -0,0 +1,65 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.io.IOException;
import java.util.List;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.InvalidFamilyOperationException;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.catalog.MetaEditor;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.util.Bytes;
/**
* Handles adding a new family to an existing table.
*/
public class TableModifyFamilyHandler extends TableEventHandler {
private final HColumnDescriptor familyDesc;
public TableModifyFamilyHandler(byte[] tableName,
HColumnDescriptor familyDesc, Server server,
final MasterServices masterServices) throws IOException {
super(EventType.C2M_MODIFY_FAMILY, tableName, server, masterServices);
this.familyDesc = familyDesc;
}
@Override
protected void handleTableOperation(List<HRegionInfo> regions) throws IOException {
HTableDescriptor htd = regions.get(0).getTableDesc();
byte [] familyName = familyDesc.getName();
if(!htd.hasFamily(familyName)) {
throw new InvalidFamilyOperationException("Family '" +
Bytes.toString(familyName) + "' doesn't exists so cannot be modified");
}
for(HRegionInfo hri : regions) {
// Update the HTD
hri.getTableDesc().addFamily(familyDesc);
// Update region in META
MetaEditor.updateRegionInfo(this.server.getCatalogTracker(), hri);
// Update region info in FS
this.masterServices.getMasterFileSystem().updateRegionInfo(hri);
}
}
}

36
src/main/java/org/apache/hadoop/hbase/master/handler/TotesHRegionInfo.java Normal file
View File

@ -0,0 +1,36 @@
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.handler;
import java.beans.EventHandler;
import org.apache.hadoop.hbase.HRegionInfo;
/**
* Implementors tote an HRegionInfo instance.
* This is a marker interface that can be put on {@link EventHandler}s that
* have an {@link HRegionInfo}.
*/
public interface TotesHRegionInfo {
/**
* @return HRegionInfo instance.
*/
public HRegionInfo getHRegionInfo();
}

Some files were not shown because too many files have changed in this diff Show More