(except TestHClient and HBaseShell) have been converted to use the new client
side objects (HTable/HBaseAdmin/HConnection) instead of HClient.
HBaseAdmin
- Expose connection methods getMaster, isMasterRunning and listTables
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@562041 13f79535-47bb-0310-9956-ffa450edef68
Modified:
HConstants
static final Text[] COL_REGIONINFO_ARRAY = new Text [] {COL_REGIONINFO};
static final Text EMPTY_START_ROW = new Text();
HMaster
- don't process a region server exit message if the lease has timed
out. Otherwise we end up with two pending server shutdown messages
to process and chaos ensues.
- don't reassign the root region when the server's lease expires. The
lease expiration handler will queue a PendingServerShutdown
operation that must run before the root region is reassigned because
the HLog of the dead server must be split before any regions served
by the dead server are reassigned.
- added some additional debug level logging
HBaseClusterTestCase
- call HConnectionManager.deleteConnection(conf) in tearDown() so that
multiple tests can be run from the same test class.
TestScanner2
- changes to make test compatible with the change from inner class
HClient.RegionLocation to public class HRegionLocation
Leases
- cancelLease just returns if the lease is not found instead of
throwing an IOException
New:
HConnection - an interface that describes the operations performed by
a connection implementation
HConnectionManager - manages connections for multiple HBase instances
and returns an object that implements HConnection from its static
method getConnection
HBaseAdmin - the HBase administrative methods refactored out of
HClient. Each HBaseAdmin object can control a single HBase
instance. To manipulate multiple instances, create multiple HBaseAdmin
objects.
HTable - The data manipulation methods refactored out of HClient. Each
HTable object talks to a single table in a single HBase
instance. Create multiple HTable objects to use more than one table.
HRegionLocation - an inner class refactored out of HClient. Each
HRegionLocation has an HRegionInfo object and an HServerAddress
object.
HClient - totally re-implemented in terms of the new classes
above. HClient is now deprecated.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@561935 13f79535-47bb-0310-9956-ffa450edef68
Detailed changes:
MiniHBaseCluster
- rewrite abortRegionServer, stopRegionServer - they now remove the
server from the map of servers.
- rewrite waitOnRegionServer - now removes thread from map of threads
TestCleanRegionServerExit
- reduce Hadoop ipc client timeout and number of retries
- use rewritten stopRegionServer and waitOnRegionServer from MiniHBaseCluster
- add code to verify that failover worked
- moved testRegionServerAbort to separate test file
TestRegionServerAbort
- new test. Uses much the same code as TestCleanRegionServerExit but
aborts the region server instead of shutting it down
cleanly. Includes code to verify that failover worked.
hbase-site.xml (in src/contrib/hbase/src/test)
- reduce master lease timeout and time between lease timeout checks so
that tests will run quicker.
HClient
- Major restructing of code that determines what region server to
contact for a specific region. The main method findServersForTable
is now recursive so that it will find the meta and root regions if
they have not already been located or will re-find them if they have
been reassigned and the old server can no longer be contacted.
- re-ordered administrative and general purpose methods so they are no
longer located in seemingly random order.
- re-ordered code in ClientScanner.loadRegions so that if the location
of the region changes, it will actually try to connect to the new
server rather than continually trying to use the connection to the
old server.
HLog
- use HashMap<Text, SequenceFile.Writer> instead of
TreeMap<Text, SequenceFile.Writer> because the TreeMap would return
a value for a key it did not have (it was the value of another
key). I have observed this before when the key is Text, but could
not create a simple test case that reproduced the problem.
- added some new DEBUG level logging
- removed call to rollWriter() from closeAndDelete(). We don't need to
start a new writer if we are closing the log.
HLogKey
- cleaned up per HADOOP-1466 (I initially modified it to add some
debug logging which was later removed, but when I was making the
modifications I took the opportunity to clean up the file)
- changed toString() format
HMaster
- better handling of RemoteException
- modified BaseScanner
- now knows if it is scanning the root or a meta region
- scanRegion no longer returns a value
- if scanning the root region, it counts the number of meta regions
it finds and sets a new AtomicInteger, numberOfMetaRegions when the
scan is complete.
- added abstract methods initialScan and maintenanceScan this allowed
run method to be implemented in the base class.
- boolean rootScanned is now volatile
- modified RootScanner
- moved actual scan into private method for readability (scanRoot)
- implementation of abstract methods just call scanRoot
- add constructor for inner static class MetaRegion
- use a BlockingQueue to queue up work for the MetaScanner
- clean up handling of an unexpected region server exit
- PendingOperation.process now returns a boolean so that HMaster.run
can determine if the operation completed or needs to be retried later
- PendingOperation processing no longer does a wait inside the process
method since this might cause a deadlock if the current operation is
waiting for another operation that has yet to be processed
HMsg
- removed MSG_REGIONSERVER_STOP_IN_ARRAY, MSG_NEW_REGION
- added MSG_REPORT_SPLIT
HRegionServer
- changed reportSplit to contain old region and new regions
- use IP from default interface rather than host name
- abort calls HLog.close() instead of HLog.rollWriter()
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@559819 13f79535-47bb-0310-9956-ffa450edef68
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ClearCommand.java
Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HBaseShell.jj
Add 'clear' command handling.
(clearCommand): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/Parser.java
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParserConstants.java
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParserTokenManager.java
Add 'clear' command handling (Generated by javacc).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpContents.java
Add 'CLEAR' help.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpManager.java
Clear screen before outputting version string.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@556356 13f79535-47bb-0310-9956-ffa450edef68
Change format of region names from TABLENAME_STARTROW_ENDROW-RANDOMID
to TABLENAME,STARTROW,ENDROW-RANDOMID. Makes it so lone table name will
sort before any region of said table.
M src/contrib/hbase/src/test/hbase-site.xml
(hbase.client.retries.number): Removed. Wasdefault value for this property.
(hbase.master.meta.thread.rescanfrequency, hbase.server.thread.wakefrequency,
hbase.regionserver.handler.count): Add values that are less than
default so unit tests are even more responsive (and finished quicker).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestToString.java
Change test so it expects region info name that has ',' delimiters
rather than '_' delimiters.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestTable.java
Rename testTable as testCreateTable.
(testTableNameClash): Test for this issue.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionInfo.java
Change format of region names so delimiter is ',' rather than '_'.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@556348 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestTable.java
(testTable): Add checking of actual exceptions thrown and
assertions that we are getting right behavior. Add a test
that has ten clients concurrently trying to create same table.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HClient.java
javadoc edit. Fix debug message that could give impression
table was found when it wasn't.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HTableDescriptor.java
Added comment on table nameing.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
(createTable): Refactored. Bulk moved to a private override.
Changed how check for existance is done.
M rc/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionInfo.java
(getTableNameFromRegionName): Utility method added.
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/TableExistsException.java
Added.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@556334 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/NOTICE.txt
Add notice of udanax contributions.
Msrc/contrib/hbase/conf/hbase-default.xml
(hbaseshell.jline.bell.enabled): Added.
M src/contrib/hbase/CHANGES.txt
(hadoop-1375) Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/package.html
Add note on how to start up hbase shell
M src/contrib/hbase/bin/hbase
Add 'shell'. Remove 'client' (shell does what it used do and more).
Removed all reader and logreader until better developed. Starting
up a reader or logreader on a running hbase system could do damage).
M src/contrib/hbase/build.xml
Add a javacc target to generate content of shell/generated subpackage.
A src/contrib/hbase/src/test/org/apache/hadoop/hbase/shell/TestHBaseShell.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/Shell.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/DeleteCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/CreateCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/DropCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/InsertCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/CommandFactory.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpContents.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ExitCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ConsoleTable.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/DescCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/SelectCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/Command.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ShowCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/BasicCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpManager.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ReturnMsg.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HBaseShell.jj
Added.
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/Token.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/TokenMgrError.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/SimpleCharStream.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParserTokenManager.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParseException.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParserConstants.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/Parser.java
Added javacc generated files.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@555415 13f79535-47bb-0310-9956-ffa450edef68
Adds a row/column filter interface and two implementations: A pager and a
row/column-value regex filter.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionInterface.java
(openScanner): Add override that specifies a row fliter.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HClient.java
(obtainScanner): Add override that specifies a row fliter.
(ColumnScanner): Add filter parameter to constructor.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
(getScanner): Add override with filter parameter.
(next): Add handling of filtering.
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/filter/InvalidRowFilterException.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/filter/RegExpRowFilter.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/filter/RowFilterSet.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/filter/PageRowFilter.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/filter/RowFilterInterface.java
Row-filter interface, exception and implementations.
A src/contrib/hbase/src/test/org/apache/hadoop/hbase/filter/TestRegExpRowFilter.java
A src/contrib/hbase/src/test/org/apache/hadoop/hbase/filter/TestPageRowFilter.java
Simple pager and regex filter tests.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@553620 13f79535-47bb-0310-9956-ffa450edef68
On shutdown, region servers and masters were just cancelling leases
without letting 'lease expired' code run -- code to clean up
outstanding locks in region server. Outstanding read locks were
getting in the way of region server getting necessary write locks
needed for the shutdown process. Also, cleaned up messaging around
shutdown so its clean -- no timeout messages as region servers try
to talk to a master that has already shutdown -- even when region
servers take their time going down.
M src/contrib/hbase/conf/hbase-default.xml
Make region server timeout 30 seconds instead of 3 minutes.
Clients retry anyways. Make so its likely region servers report
in their shutdown message before their lease expires on master.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/Leases.java
(closeAfterLeasesExpire): Added.
* src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Added comments.
(stop): Converted from public to default access (master shuts
down regionservers).
(run): Use leases.closeAfterLeasesExpire instead of leases.close.
Changed log of main thread exit from DEBUG to INFO.
* src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
(letRegionsServersShutdown): Add better explaination of shutdown
process to method doc. Changed timeout waits from
hbase.regionserver.msginterval to threadWakeFrequency.
(regionServerReport): If closing, we used to immediately respond
to region server with a MSG_REGIONSERVER_STOP. This meant that
we avoided handling of the region servers MSG_REPORT_EXITING sent
on shutdown so region servers had no chance to cancel their lease
in the master. Reordered. Moved sending of MSG_REGIONSERVER_STOP
to after handling of MSG_REPORT_EXITING. Also, in handling of
MSG_REGIONSERER_STOP removed cancelling of leases. Let leases
expire normally (or get cancelled when the region server comes in
with MSG_RPORT_EXITING).
* src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMsg.java
(MSG_REGIONSERVER_STOP_IN_ARRAY): Added.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@552376 13f79535-47bb-0310-9956-ffa450edef68
AbstractMergeTestBase, HBaseTestCase: move createNewHRegion to HBaseTestCase
MiniHBaseCluster: add deleteOnExit, getDFSCluster, fix Javadoc
TestScanner2: moved KeyedData to org.apache.hadoop.hbase.io
TestTableMapReduce: new test case to test map/reduce interface to HBase
hbase-site.xml: change hbase.client.pause from 3 to 5 seconds, hbase.client.retries.number to 5 so that tests will not time out or run out of retries
HClient: moved KeyedData to org.apache.hadoop.hbase.io, fix javadoc, add method getStartKeys
HMaster: moved KeyedData to org.apache.hadoop.hbase.io, remove unused variables, remove extraneous throws clause,
HRegionInterface, HRegionServer: moved KeyedData to org.apache.hadoop.hbase.io
KeyedData: moved KeyedData to org.apache.hadoop.hbase.io
KeyedDataArrayWritable: new class to support HBase map/reduce
org.apache.hadoop.hbase.mapred: new classes for map/reduce
- GroupingTableMap
- IdentityTableMap
- IdentityTableReduce
- TableInputFormat
- TableMap
- TableOutputCollector
- TableOutputFormat
- TableReduce
- TableSplit
hbase/bin/hbase: changes for map/reduce
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@552127 13f79535-47bb-0310-9956-ffa450edef68
* src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMemcache.java
(findFirstRow): Change compare from startsWith -- looking for
an 'exact' match -- to instead use compareTo and if >= 0, then
we are at first key.
* src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Add logging of problematic directory if server won't start because
log file exists.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@551039 13f79535-47bb-0310-9956-ffa450edef68
so that region servers are assigned the regions have a log to apply edits from. Enhance fail over
capabilities.
For all the files modified, clean up javadoc, class method and field visibility.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@546192 13f79535-47bb-0310-9956-ffa450edef68
Jim Kellerman
Michael Stack
Doug Cutting